Systems and methods for providing digital health services

ABSTRACT

The present disclosure is directed to providing digital health services. In some embodiments, systems and methods for conducting virtual or remote sessions between patients and clinicians are disclosed. During the sessions, media content (e.g., images, video content, audio content, etc.) may be captured as the patient performs one or more tasks. The media content may be presented to the clinician and used to evaluate a condition of the patient or a state of the condition, adjust treatment parameters, provide therapy, or other operations to treat the patient. The analysis of the media content may be aided by one or more machine learning/artificial intelligence models that analyze various aspects of the media content, augment the media content, or other functionality to aid in the treatment of the patient.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority of U.S. ProvisionalPatent Application No. 63/234,646, filed Aug. 18, 2021, U.S. ProvisionalPatent Application No. 63/297,176, filed Jan. 6, 2022, and U.S.Provisional Patent Application No. 63/311,031, filed Feb. 16, 2022,which are hereby incorporated by reference in their entirety.

TECHNICAL FIELD

The present application is generally directed to providing digitalhealth services to patients.

BACKGROUND

Implantable medical devices have changed how medical care is provided topatients having a variety of chronic illnesses and disorders. Forexample, implantable cardiac devices improve cardiac function inpatients with heart disease by improving quality of life and reducingmortality rates. Respective types of implantable neurostimulatorsprovide a reduction in pain for chronic pain patients and reduce motordifficulties in patients with Parkinson's disease and other movementdisorders. A variety of other medical devices are proposed and are indevelopment to treat other disorders in a wide range of patients.

Many implantable medical devices and other personal medical devices areprogrammed by a physician or other clinician to optimize the therapyprovided by a respective device to an individual patient. Typically, theprogramming occurs using short-range communication links (e.g.,inductive wireless telemetry) in an in-person or in-clinic setting.Since such communications typically require close immediate contact,there is only an extremely small likelihood of a third-partyestablishing a communication session with the patient's implanted devicewithout the patient's knowledge.

Remote patient care is a healthcare delivery method that aims to usetechnology to provide patient health outside of a traditional clinicalsetting (e.g., in a doctor's office or a patient's home). It is widelyexpected that remote patient care may increase access to care anddecrease healthcare delivery costs.

SUMMARY

The present application is generally directed to providing digitalhealth services to patients.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present disclosure are illustrated by way of example,and not by way of limitation, in the Figures of the accompanyingdrawings in which like references indicate similar elements. It shouldbe noted that different references to “an” or “one” embodiment in thisdisclosure are not necessarily to the same embodiment, and suchreferences may mean at least one. Further, when a particular feature,structure, or characteristic is described in connection with anembodiment, it is submitted that it is within the knowledge of oneskilled in the art to effectuate such feature, structure, orcharacteristic in connection with other embodiments whether or notexplicitly described.

The accompanying drawings are incorporated into and form a part of thespecification to illustrate one or more exemplary embodiments of thepresent disclosure. Various advantages and features of the disclosurewill be understood from the following Detailed Description taken inconnection with the appended claims and with reference to the attacheddrawing Figures in which:

FIG. 1A depicts an example architecture of a system configured tosupport remote patient therapy as part of an integrated remote careservice session in a virtual clinic environment that may be deployed ina cloud-centric digital health implementation according to one or moreembodiments of the present patent disclosure;

FIG. 1B depicts an example network environment wherein the remote careservice architecture of FIG. 1A may be implemented according to arepresentative embodiment;

FIG. 2 depicts a flowchart illustrative of blocks, steps and/or actsthat may be (re)combined in one or more arrangements with or withoutadditional flowcharts of the present disclosure for facilitating remotecare therapy in a secure network environment for purposes of someembodiments;

FIG. 3 depicts a flowchart illustrative of blocks, steps and/or actsthat may be implemented for establishing a communication session with animplantable medical device;

FIGS. 4A and 4B depicts flowcharts illustrative of a remote carescenario involving an example digital health network architecturewherein an integrated remote care session may be established between apatient and a clinician operating respective controller devices forpurposes of some embodiments of the present disclosure;

FIGS. 5A and 5B depict representations of an example user interface andassociated dialog boxes provided with a clinician programmer device forselecting different therapy applications and/or service modes forpurposes of some embodiments of the present disclosure;

FIG. 6 depicts a representation of an example user interface providedwith a clinician programmer device for facilitating controls withrespect to an AV communication session and a remote therapy session inan integrated remote care service application for purposes of someembodiments of the present disclosure;

FIG. 7 depicts a block diagram of an external device that may beconfigured as a clinician programmer device, a patient controller deviceor an authorized third-party device operative in a digital healthnetwork architecture for purposes of some embodiments of the presentdisclosure;

FIG. 8 depicts a block diagram illustrating additional details of apatient controller device operative in a digital health networkarchitecture for purposes of some embodiments of the present disclosure;

FIG. 9 depicts a block diagram illustrating additional details of aclinician programmer device operative in a digital health networkarchitecture for purposes of some embodiments of the present disclosure;

FIG. 10 depicts a block diagram of an IMD and associated system that maybe configured for facilitating a remote care therapy application and/ora local therapy session for purposes of an example embodiment of thepresent disclosure;

FIGS. 11A-11H depict representations of an example user interface andassociated dialog boxes provided with a patient controller device forselecting different therapy applications and/or service modes and forfacilitating controls with respect to an AV communication session and aremote therapy session in an integrated remote care service applicationfor purposes of an embodiment of the present disclosure;

FIG. 12 depicts an example cloud-centric digital healthcare networkarchitecture including one or more virtual clinic platforms, patientreport processing platforms and remote data logging platforms forpurposes of some example embodiments of the present disclosure;

FIG. 13 a flowchart depicting operations that may occur during a virtualclinic or remote programming session according to embodiments of thepresent disclosure;

FIG. 14 is a flowchart depicting an exemplary flow of operations forcreating a predictive disorder model using ML/AI processing according tosome embodiments;

FIG. 15 depicts aspects of ML/AI learning data sets and predictivemodels according to embodiments;

FIG. 16 is a flowchart depicting exemplary operations for processingvideo data to support provision of neurostimulation therapies to patientaccording to some embodiments;

FIG. 17A-17B depict sets of key landmark points that may be utilized forfacial expression analysis and pose analysis according to someembodiments;

FIG. 18A-18D show screenshots illustrating exemplary aspects of userinterfaces utilized to control and view information during a virtual orremote session conducted according to some embodiments;

FIG. 19 is a flowchart depicting exemplary operations for conducting avirtual clinic/remote programming session according to some embodiments;

FIG. 20 is a flowchart depicting exemplary operations for conducting aphysical therapy session in accordance with some embodiments; and

FIG. 21 is a screenshot of an exemplary graphical user interface forpresenting performance data associated with a patient.

The foregoing has outlined rather broadly the features and technicaladvantages of the present invention in order that the detaileddescription of the invention that follows may be better understood.Additional features and advantages of the invention will be describedhereinafter which form the subject of the claims of the invention. Itshould be appreciated by those skilled in the art that the conceptionand specific embodiment disclosed may be readily utilized as a basis formodifying or designing other structures for carrying out the samepurposes of the present invention. It should also be realized by thoseskilled in the art that such equivalent constructions do not depart fromthe spirit and scope of the invention as set forth in the appendedclaims. The novel features which are believed to be characteristic ofthe invention, both as to its organization and method of operation,together with further objects and advantages will be better understoodfrom the following description when considered in connection with theaccompanying figures. It is to be expressly understood, however, thateach of the figures is provided for the purpose of illustration anddescription only and is not intended as a definition of the limits ofthe present invention.

DETAILED DESCRIPTION

In the description herein for embodiments of the present disclosure,numerous specific details are provided, such as examples of circuits,devices, components and/or methods, to provide a thorough understandingof embodiments of the present disclosure. One skilled in the relevantart will recognize, however, that an embodiment of the disclosure can bepracticed without one or more of the specific details, or with otherapparatuses, systems, assemblies, methods, components, materials, parts,and/or the like set forth in reference to other embodiments herein. Inother instances, well-known structures, materials, or operations are notspecifically shown or described in detail to avoid obscuring aspects ofembodiments of the present disclosure. Accordingly, it will beappreciated by one skilled in the art that the embodiments of thepresent disclosure may be practiced without such specific components. Itshould be further recognized that those of ordinary skill in the art,with the aid of the Detailed Description set forth herein and takingreference to the accompanying drawings, will be able to make and use oneor more embodiments without undue experimentation.

Additionally, terms such as “coupled” and “connected,” along with theirderivatives, may be used in the following description, claims, or both.It should be understood that these terms are not necessarily intended assynonyms for each other. “Coupled” may be used to indicate that two ormore elements, which may or may not be in direct physical or electricalcontact with each other, co-operate or interact with each other.“Connected” may be used to indicate the establishment of communication,i.e., a communicative relationship, between two or more elements thatare coupled with each other. Further, in one or more example embodimentsset forth herein, generally speaking, an electrical element, componentor module may be configured to perform a function if the element may beprogrammed for performing or otherwise structurally arranged to performthat function.

Example embodiments described herein relate to aspects ofimplementations of an integrated digital health network architecturethat may be effectuated as a convergence of various technologiesinvolving diverse end user devices and computing platforms,heterogeneous network connectivity environments, agile software as amedical device (SaMD) deployments, data analytics and machine learning,secure cloud-centric infrastructures for supporting remote healthcare,etc. Some embodiments may be configured to support various types ofhealthcare solutions including but not limited to remote patientmonitoring, integrated session management for providing telehealthapplications as well as remote care therapy applications, personalizedtherapy based on advanced analytics of patient and clinician data,remote trialing of neuromodulation therapies, e.g., painmanagement/amelioration solutions, and the like. Whereas some exampleembodiments may be particularly set forth with respect to implantablepulse generator (IPG) or neuromodulator systems for providing therapy toa desired area of a body or tissue based on a suitable stimulationtherapy application, such as spinal cord stimulation (SCS) systems orother neuromodulation systems, it should be understood that exampleembodiments disclosed herein are not limited thereto but have broadapplicability. Some example remote care therapy applications maytherefore involve different types of implantable devices such asneuromuscular stimulation systems and sensors, dorsal root ganglion(DRG) stimulation systems, deep brain stimulation systems, cochlearimplants, retinal implants, implantable cardiac rhythm managementdevices, implantable cardioverter defibrillators, pacemakers, and thelike, as well as implantable drug delivery/infusion systems, implantabledevices configured to effectuate real-time measurement/monitoring of oneor more physiological functions of a patient's body (i.e., patientphysiometry), including various implantable biomedical sensors andsensing systems. Further, whereas some example embodiments of remotecare therapy applications may involve implantable devices, additionaland/or alternative embodiments may involve external personal devicesand/or noninvasive/minimally invasive (NIMI) devices (e.g., wearablebiomedical devices, transcutaneous/subcutaneous devices, etc.) that maybe configured to provide therapy to the patients analogous to theimplantable devices. Accordingly, all such devices may be broadlyreferred to as “personal medical devices,” “personal biomedicalinstrumentation,” or terms of similar import, at least for purposes ofsome example embodiments of the present disclosure.

As used herein, a network element, platform or node may be comprised ofone or more pieces of network equipment, including hardware and softwarethat communicatively interconnects other equipment on a network (e.g.,other network elements, end stations, etc.), and is adapted to host oneor more applications or services, more specifically healthcareapplications and services, with respect to a plurality of end users(e.g., patients, clinicians, respective authorized agents, andassociated client devices) as well as other endpoints such as medical-and/or health-oriented Internet of Medical Things (IoMT) devices/sensorsand/or other Industrial IoT-based entities. As such, some networkelements may be operatively disposed in a cellular wireless or satellitetelecommunications network, or a broadband wireline network, whereasother network elements may be disposed in a public packet-switchednetwork infrastructure (e.g., the Internet or worldwide web, alsosometimes referred to as the “cloud”), private packet-switched networkinfrastructures such as Intranets and enterprise networks, as well asservice provider network infrastructures, any of which may span orinvolve a variety of access networks, backhaul and core networks in ahierarchical arrangement. In still further arrangements, one or morenetwork elements may be disposed in cloud-based platforms or datacentershaving suitable equipment running virtualized functions or applications,which may be configured for purposes of facilitating patient monitoring,remote therapy, other telehealth/telemedicine applications, etc. forpurposes of one or more example embodiments set forth hereinbelow.

One or more embodiments of the present patent disclosure may beimplemented using different combinations of software, firmware, and/orhardware. Thus, one or more of the techniques shown in the Figures(e.g., flowcharts) may be implemented using code and data stored andexecuted on one or more electronic devices or nodes (e.g., a subscriberclient device or end station, a network element, etc.). Such electronicdevices may store and communicate (internally and/or with otherelectronic devices over a network) code and data using computer-readablemedia, such as non-transitory computer-readable storage media (e.g.,magnetic disks, optical disks, random access memory, read-only memory,flash memory devices, phase-change memory, etc.), transitorycomputer-readable transmission media (e.g., electrical, optical,acoustical or other form of propagated signals—such as carrier waves,infrared signals, digital signals), etc. In addition, such networkelements may typically include a set of one or more processors coupledto one or more other components, such as one or more storage devices(e.g., non-transitory machine-readable storage media) as well as storagedatabase(s), user input/output devices (e.g., a keyboard, a touchscreen, a pointing device, and/or a display), and network connectionsfor effectuating signaling and/or bearer media transmission.

Without limitation, an example cloud-centric digital healthcare networkarchitecture involving various network-based components, subsystems,service nodes etc., as well as myriad end user deployments concerningpatients, clinicians and authorized third-party agents is illustrated inFIG. 12 wherein one or more embodiments of the present patent disclosuremay be practiced. In one arrangement, example architecture 1260 mayinclude one or more virtual clinic platforms 1214, remote data loggingplatforms 1216, patient/clinician report processing platforms 1218, aswell as data analytics platforms 1220 and security platforms 1222, atleast some of which may be configured and/or deployed as an integrateddigital health infrastructure 1212 for effectuating some exampleembodiments of the present disclosure. One or more pools of patientshaving a myriad of health conditions and/or receiving assortedtreatments, who may be geographically distributed in various locations,areas, regions, etc., are collectively shown at reference numeral 1204,wherein individual patients may be provided with one or more suitableIMDs/IPGs, NIMI devices, other personal biomedical instrumentation,etc., depending on respective patients' health conditions and/ortreatments. A plurality of clinician programmer devices 1208, patientcontroller devices 1210, and authorized third-party devices 1211associated with respective users (e.g., clinicians, medicalprofessionals, patients and authorized agents thereof) may be deployedas external devices 1206 that may be configured to interact withpatients' IMDs and/or NIMI devices for effectuating therapy, monitoring,data logging, secure file transfer, etc., via local communication pathsor over network-based remote communication paths established inconjunction with the digital health infrastructure network 1212.

Clinician controller device 1208 may permit programming of IPG 170 toprovide a number of different stimulation patterns or therapies to thepatient as appropriate for a given patient and/or disorder. Examples ofdifferent stimulation therapies include conventional tonic stimulation(continuous train of stimulation pulses at a fixed rate), BurstDRstimulation (burst of pulses repeated at a high rate interspersed withquiescent periods with or without duty cycling), “high frequency”stimulation (e.g., a continuous train of stimulation pulses at 10,000Hz), noise stimulation (series of stimulation pulses with randomizedpulse characteristics such as pulse amplitude to achieve a desiredfrequency domain profile). Any suitable stimulation pattern orcombination thereof can be provided by IPG 170 according to someembodiments. Controller device 1208 communicates the stimulationparameters and/or a series of pulse characteristics defining the pulseseries to be applied to the patient to IPG 170 to generate the desiredstimulation therapy.

IPG 170 may be adapted to apply a variety of neurostimulation therapieswhile controller device 1208 may send signals to IPG 170 related to suchtherapies. Examples of suitable therapies include tonic stimulation (inwhich a fixed frequency pulse train) is generated, burst stimulation (inwhich bursts of multiple high frequency pulses) are generated which inturn are separated by quiescent periods, “high frequency” stimulation,multi-frequency stimulation, and noise stimulation. Descriptions ofrespective neurostimulation therapies are provided in the followingpublications: (1) Schu S., Slotty P. J., Bara G., von Knop M., Edgar D.,Vesper J. A Prospective, Randomised, Double-blind, Placebo-controlledStudy to Examine the Effectiveness of Burst Spinal Cord StimulationPatterns for the Treatment of Failed Back Surgery Syndrome.Neuromodulation 2014; 17: 443-450; (2) AI-Kaisy A1, Van Buyten J P, SmetI, Palmisani S, Pang D, Smith T. 2014. Sustained effectiveness of 10 kHzhigh-frequency spinal cord stimulation for patients with chronic, lowback pain: 24-month results of a prospective multicenter study. PainMed. 2014 March; 15(3):347-54; and (3) Sweet, Badjatiya, Tan D1, Miller.Paresthesia-Free High-Density Spinal Cord Stimulation forPostlaminectomy Syndrome in a Prescreened Population: A Prospective CaseSeries. Neuromodulation. 2016 April; 19(3):260-7. Noise stimulation isdescribed in U.S. Pat. No. 8,682,44162. Burst stimulation is describedin U.S. Pat. No. 8,224,453 and U.S. Published Application No.20060095088. A “coordinated reset” pulse pattern is applied to neuronalsubpopulation/target sites to desynchronize neural activity in thesubpopulations. Coordinated reset stimulation is described, for example,by Peter A. Tass et al in COORDINATED RESET HAS SUSTAINED AFTER EFFECTSIN PARKINSONIAN MONKEYS, Annals of Neurology, Volume 72, Issue 5, pages816-820, November 2012, which is incorporated herein by reference. Theelectrical pulses in a coordinated reset pattern are generated in burstsof pulses with respective bursts being applied to tissue of the patientusing different electrodes in a time-offset manner. The time-offset isselected such that the phase of the neural-subpopulations are reset in asubstantially equidistant phase-offset manner. By resetting neuronalsubpopulations in this manner, the population will transition to adesynchronized state by the interconnectivity between the neurons in theoverall neuronal population. All of these references are incorporatedherein by reference.

In one arrangement, example architecture 1260 may encompass ahierarchical/heterogeneous network arrangement comprised of one or morefronthaul radio access network (RAN) portions or layers, one or morebackhaul portions or layers, and one or more core network portions orlayers, each of which may in turn include appropriate telecommunicationsinfrastructure elements, components, etc., cooperatively configured foreffectuating a digital healthcare ecosystem involving patients' IMDsand/or NIMI devices 1204, external devices 1206, and one or morecomponents of the digital health infrastructure network 1212, wherein atleast a portion of the components of the infrastructure network 1212 maybe operative as a cloud-based system for purposes of some embodimentsherein. Further, at least a portion of the components of the digitalhealth infrastructure network 1212 operating as a system 1200, one ormore patients' IMDs and/or NIMI devices 1204, and one or more externaldevices 1206 may be configured to execute suitable medical/healthsoftware applications in a cooperative fashion (e.g., in a server-clientrelationship) for effectuating various aspects of remote patientmonitoring, telemedicine/telehealth applications, remote care therapy,etc. Without limitation, example embodiments of the present disclosuremay relate to one or more aspects set forth immediately below.

In some example arrangements, a virtual clinic may be configured toprovide patients and/or clinicians the ability to perform remotetherapies using a secure telehealth session. To enhance clinicianinteraction and evaluation of a patient during a secure telehealthsession, example embodiments herein may be configured to providespecific user interface (UI) layouts and controls for clinicianprogrammer devices and/or patient controller devices for facilitatingreal-time kinematic and/or auditory data analysis, which may beaugmented with suitable artificial intelligence (AI) and/or machinelearning (ML) techniques (e.g., neural networks, etc.) in somearrangements. Further, some example embodiments with respect to theseaspects may involve providing kinematic UI settings that enabledifferent types of overlays (e.g., with or without a pictorialrepresentation of the patient). Some example embodiments may beconfigured to enable one or more of the following features andfunctionalities: (i) separate or combined audio and/or peripheral sensorstreams; (ii) capture of assessments from separate or differentcombinations of body features such as, for example, limbs, hands, face,etc.; (iii) replay of another clinician's video including the patient'skinematic analysis (e.g., a secondary video stream with patient data),and the like.

In some arrangements, video-based real-time kinematic analysis mayemploy statistical methods within a pipeline of modeling techniques totrack inter-frame correlation of images. To maintain real-time operationwithin a fixed compute environment such as an edge device (e.g., apatient controller device, a clinician programmer device, etc.), theinference latency needs to be less than certain thresholds in order toachieve reliable and/or acceptable performance. Some example embodimentsherein may therefore relate to providing a scheme for improving theaccuracy of real-time kinematic and/or auditory analysis based oncontext-aware dynamic (re)configuration of a neural network model orengine trained for facilitating real-time kinematic and/or auditory dataanalysis.

In some arrangements involving neurostimulation therapy, differentstimulation settings and/or programs may be configured for providingvaried levels of comfort to the patients, wherein respective patientsmay likely need to change individual settings depending on a number offactors (e.g., time of day, type(s) and/or level(s) of activities ortasks being engaged by the patients, and the like). Further, continueduse of a stimulation program or setting over an extended period of timecould result in habituation that may reduce the benefits of therapy.Some example embodiments herein may therefore relate to a system andmethod for providing recommendations/reconfigurations of programsettings based on the patient's usage of the IMD and clinicalobservations/recommendations, which may facilitate context-sensitiveselection of neuromodulation programs/settings.

In some arrangements, video-, audio-, and/or sensing-based analytics andassociated ML-based techniques effectuated using one or more constituentcomponents of the digital health infrastructure 1212 may providevaluable insights with respect to the diagnosis/or prognosis ofindividual patients, especially those having certain neurologicaldisorders. Some of the neurological symptoms may depend on the context(e.g., time of day, activity type, psychological/emotional conditions ofthe patient, etc.) such that a generalized ML model may not besufficiently accurate for predictive purposes in a particular setting.Some example embodiments herein may therefore relate to a scheme forrapidly collecting relevant patient data and analyzing/manipulating thedata for generating suitable training datasets with respect to selectML-based models using an accelerated inference approach. In someembodiments, context information is gathered passively or throughpatient self-reporting. In some embodiments, context information may begathered prior to, immediately before, or at the initiation of a remoteprogramming session. Such context information may be used for AI/MLprocessing of patient condition as discussed herein during a remoteprogramming session. For example, the context information may be used toimprove the accuracy to AI/ML trained models for detecting neurologicalconditions (pain level, tremor, rigidity, and/or the like).

Because the video/audio data collected and used for training, validatingand testing AI/ML-based models can contain various pieces of personalidentification/identifiable information (PII) indicators associated withpatients, it is appropriate to protect the privacy of the patient databy providing a suitable end-to-end security architecture. In somerelated arrangements, one or more constituent components of the digitalhealth infrastructure 1212 may therefore be configured to facilitatesecure transfer with respect to the patient data collected for purposesof data analytics and associated ML-based techniques as set forthherein. Some example embodiments in this regard may be configured toprovide a scheme for de-identifying data associated with neuromodulationpatients, e.g., either in real-time or in a post-processed environment,that still allows implementing ML-based techniques and data sharing in asecure cooperative arrangement. Still further embodiments may relate tofacilitating an improved method of removal of identifiable informationfrom video/audio data streams generated pursuant to a therapy session,e.g., a remote therapy session.

In relation to certain aspects of telemedicine, it is recognized thatmonitoring body rigidity may be an important factor in certain types ofmotor/neurological disorders, e.g., Parkinson's disease (PD). Rigiditymay be defined as an involuntary increase in muscle tone in certainportions of the patient body, e.g., generally affecting arms, neck, leg,hip or trunk of the patient. Rigidity can be classified as lead-pipewhen the movement is smooth and consistent or cog-wheeling when it isratchet-type. Cog-wheeling generally occurs when rigidity issuperimposed on tremor. Rigidity may be measured in-clinic with theclinician physically manipulating the patient to assess signs andsymptoms. In some experimental setups, rigidity can also be measuredwith an apparatus to detect the displacement for applied force. However,in a telehealth scenario, there is a need to accurately and reliablymeasure rigidity in a remote setting. Some example embodiments hereinmay therefore relate to a system and method for facilitatingremote-based assessment of rigidity that may involve combining signalsfrom sensors and video analytics obtained via a secure remote sessionwith the patient.

In still further aspects relating to remote patient monitoring, someexample embodiments may be configured to effectuate a closed-loop,sensor-based AI-driven exercise training platform that may beimplemented by way of an integrative telehealth application. Suchembodiments advantageously leverage the principle that exercise regimeninvolving balance training as part of a physical routine can provideadditional benefits for patients with balance/gait-related disorders ontop of the benefits exercising itself already brings. Whereas exercisesthat involve balance training may be taught by a teacher/instructor in aface-to-face setting, where the teacher can manipulate the trainee'sgesture in addition to offering visual and verbal instructions, exampleembodiments herein may involve a remote learning and real-time patientmonitoring session for facilitating an AI-driven, network-based remoteexercising arrangement. Additionally, where patients with movementdisorders such as PD often report difficulties with everyday tasks suchas buttoning, brushing, writing, etc., example embodiments may beconfigured to provided individualized training tailored to the patientbased on AI integration in order to enable the experience of a personaltrainer with focused attention and real-time corrective measures forgesture training.

Additional details with respect to the various constituent components ofthe digital health infrastructure 1212, example external devices 1206comprising clinician programmer devices 1208, patient controller devices1210 and/or third-party devices 1211, as well as various interactionsinvolving the network-based entities and the end points (also referredto as edge devices) will be set forth immediately below in order toprovide an example architectural framework wherein one or more of theforegoing embodiments may be implemented and/or augmented according tothe teachings herein.

Turning to FIG. 1A, depicted therein is an example architecture of asystem configured to support remote patient therapy as part of anintegrated remote care service session in a virtual clinic environmentthat may be deployed in a cloud-centric digital health implementationfor purposes of an embodiment of the present patent disclosure. As usedherein, a “remote care system” may describe a healthcare delivery systemconfigured to support a remote care service over a network in acommunication session between a patient and a clinician whereintelehealth or telemedicine applications involving remote medicalconsultations as well as therapy applications involving remoteprogramming of the patient's IMD may be launched via a unifiedapplication interface facilitated by one or more network entities (e.g.,as a virtual clinic platform). In some arrangements, a remote caresystem may also include a remote patient monitoring system and/or aremote healthcare provisioning system without the involvement of aclinician. In still further arrangements, a remote care system mayinclude one or more AI-based expert systems or agents, e.g., involvingsupervised learning, that may be deployed in a network to provide orotherwise augment the capabilities of a system to effectuate enhancedhealthcare solutions relating to diagnosis, remote learning, therapyselection, as well as facilitate network-based solutions for enhancingpatients' overall well-being. In some aspects, remote care therapy mayinvolve any care, programming, or therapy instructions that may beprovided by a doctor, a medical professional or a healthcare provider,and/or their respective authorized agents, collectively referred to as a“clinician”, using a suitable clinician device, with respect to thepatient's IMD, wherein such therapy instructions may be mediated,proxied or otherwise relayed by way of a controller device associatedwith the patient. As illustrated, example remote care system 100A mayinclude a plurality of patient controller devices exemplified by patientcontroller device 150 and a plurality of clinician programmer devicesexemplified by a clinician programmer device 180 (also referred to as aclinician programmer or clinician device) that may interact with anetwork-based infrastructure via respective communication interfaces.Example patient and clinician devices may each include a correspondingremote care service application module, e.g., a patient controllerapplication 152 and a clinician programmer/controller application 182,executed on a suitable hardware/software platform for supporting aremote care service that may be managed by a network entity 155. In someembodiments, example network entity 155 may comprise a distributeddatacenter or cloud-based service infrastructure (e.g., disposed in apublic cloud, a private cloud, or a hybrid cloud, involving at least aportion of the Internet) operative to host a remote care sessionmanagement service 157. In one arrangement, patient controllerapplication 152 and clinician programmer application 182 may eachinclude a respective remote session manager 154, 184 configured toeffectuate or otherwise support a corresponding communication interface160, 190 with network entity 155 using any known or heretofore unknowncommunication protocols and/or technologies. In one arrangement,interfaces 160, 190 are each operative to support an audio/video oraudiovisual (AV) channel or session 163A, 163B and a remote therapychannel or session 165A, 165B, respectively, with an AV communicationservice 161A and a remote therapy session service 161B of the remotecare session management service 157 as part of a common bi-directionalremote care session 159, 199 established therewith. In one arrangement,patient controller application 152 and clinician programmer application182 may each further include or otherwise support suitable graphicaluser interfaces (GUIs) and associated controls 156, 186, as well ascorresponding AV managers 158, 188, each of which may be interfaced withrespective remote session managers 154, 184 for purposes of one or moreembodiments of the present disclosure as will be set forth in additionaldetail further below. Remote care session manager 154 of the patientcontroller application 152 and remote care session manager 184 of theclinician programmer application 182 may each also be interfaced with acorresponding data logging manager 162, 186 for purposes of stillfurther embodiments of the present disclosure. In one arrangement,remote care session manager 154 of patient controller application 152 isfurther interfaced with a security manager 164, which may be configuredto facilitate secure or trusted communication relationships with thenetwork entity 155. Likewise, remote care session manager 184 ofclinician programmer application 182 may also be interfaced with asecurity manager 188 that may be configured to facilitate secure ortrusted communication relationships with the network entity 155. Eachsecurity manager 164, 188 may be interfaced with a corresponding therapycommunication manager 166, 190 with respect to facilitating securetherapy communications between the clinician programmer device 180 andthe patient controller device 150. Therapy communication manager 166 ofthe patient controller application 152 may also interface with a localcommunication module 168 operative to effectuate secure communicationswith the patient's IPG/IMD 170 using a suitable short-rangecommunications technology or protocol. In still further arrangements,security managers 164, 188 of patient controller and clinicianprogrammer applications 152, 182 may be configured to interface with theremote care session management service 157 to establish trustedrelationships between patient controller device 150, clinicianprogrammer device 180 and IPG/IMD 170 based on the exchange of a varietyof parameters, e.g., trusted indicia, cryptographic keys andcredentials, etc.

In one arrangement, the integrated remote care session managementservice 157 may include a session data management module 171, an AVsession recording service module 175 and a registration service module183, as well as suitable database modules 173, 185 for storing sessiondata and user registration data, respectively. In some arrangements, atleast part of the session data may include user-characterized datarelating to AV data, therapy settings data, network contextual data, andthe like, for purposes of still further embodiments of the presentpatent disclosure.

Skilled artisans will realize that example remote care systemarchitecture 100A set forth above may be advantageously configured toprovide both telehealth medical consultations as well as therapyinstructions over a communications network while the patient and theclinician/provider are not in close proximity of each other (e.g., notengaged in an in-person office visit or consultation). Accordingly, insome embodiments, a remote care service of the present disclosure mayform an integrated healthcare delivery service effectuated via a commonapplication user interface that not only allows healthcare professionalsto use electronic communications to evaluate and diagnose patientsremotely but also facilitates remote programming of the patient'sIPG/IMD for providing appropriate therapy, thereby enhancing efficiencyas well as scalability of a delivery model. Additionally, example remotecare system architecture 100A may be configured to effectuate variousother aspects relating to remote learning, remote patient monitoring,etc., as noted above. Further, an implementation of example remote caresystem architecture 100A may involve various types of networkenvironments deployed over varying coverage areas, e.g., homogenousnetworks, heterogeneous networks, hybrid networks, etc., which may beconfigured or otherwise leveraged to provide patients with relativelyquick and convenient access to diversified medical expertise that may begeographically distributed over large areas or regions, preferably viasecure communications channels in some example embodiments as will beset forth in detail further below.

FIG. 1B depicts an example network environment 100B wherein the remotecare service architecture of FIG. 1A may be implemented according tosome embodiments. Illustratively, example network environment 100B maycomprise any combination or sub-combination of a public packet-switchednetwork infrastructure (e.g., the Internet or worldwide web, alsosometimes referred to as the “cloud”, as noted above), privatepacket-switched network infrastructures such as Intranets and enterprisenetworks, health service provider network infrastructures, and the like,any of which may span or involve a variety of access networks, backhauland core networks in an end-to-end network architecture arrangementbetween one or more patients, e.g., patient(s) 102, and one or moreauthorized clinicians, healthcare professionals, or agents thereof,e.g., generally represented as caregiver(s) or clinician(s) 138. Examplepatient(s) 102, each having one or more suitable implantable and/or NIMIdevices, e.g., IMD 103, may be provided with a variety of correspondingexternal devices for controlling, programming, otherwise (re)configuringthe functionality of respective implantable device(s) 103, as is knownin the art. Such external devices associated with patient(s) 102,referred to herein as patient devices 104, which are representative ofpatient controller device 150 shown in FIG. 1A, may comprise a varietyof user equipment (UE) devices, tethered or untethered, that may beconfigured to engage in remote care sessions involving telehealth and/ortherapy sessions according to some embodiments described below. By wayof example, patient devices 104 may comprise commercial off-the-shelf(COTS) equipment or proprietary portable medical/healthcare devices(non-COTS), which may be configured to execute a therapy/digitalhealthcare application program or “app”, wherein various types ofcommunications relating to control, therapy/diagnostics, and/or devicefile management may be effectuated for purposes of some embodiments.Accordingly, example patient devices 104 may include, in addition toproprietary medical devices, devices such as smartphones, tablets orphablets, laptops/desktops, handheld/palmtop computers, wearable devicessuch as smart glasses and smart watches, personal digital assistant(PDA) devices, smart digital assistant devices, etc., any of which mayoperate in association with one or more virtual assistants, smarthome/office appliances, smart TVs, external/auxiliary AV equipment,virtual reality (VR), mixed reality (MR) or augmented reality (AR)devices, and the like, which are generally exemplified by wearabledevice(s) 106, smartphone(s) 108, tablet(s)/phablet(s) 110, computer(s)112, and AV equipment 114. As such, example patient devices 104 mayinclude various types of communications circuitry or interfaces toeffectuate wired or wireless communications, short-range and long-rangeradio frequency (RF) communications, magnetic field communications,etc., using any combination of technologies, protocols, and the like,with external networked elements and/or respective implantable devices103 corresponding to patient(s) 102. With respect to networkedcommunications, patient devices 104 may be configured, independently orin association with one or more digital/virtual assistants, smarthome/premises appliances and/or home networks, to effectuate mobilecommunications using technologies such as Global System for MobileCommunications (GSM) radio access network (GRAN) technology, EnhancedData Rates for Global System for Mobile Communications (GSM) Evolution(EDGE) network (GERAN) technology, 4G Long Term Evolution (LTE)technology, Fixed Wireless technology, 5th Generation PartnershipProject (5GPP or 5G) technology, Integrated Digital Enhanced Network(IDEN) technology, WiMAX technology, various flavors of Code DivisionMultiple Access (CDMA) technology, heterogeneous access networktechnology, Universal Mobile Telecommunications System (UMTS)technology, Universal Terrestrial Radio Access Network (UTRAN)technology, All-IP Next Generation Network (NGN) technology, as well astechnologies based on various flavors of IEEE 802.11 protocols (e.g.,WiFi), and other access point (AP)-based technologies andmicrocell-based technologies involving small cells, femtocells,picocells, etc. Further, some embodiments of patient devices 104 mayalso include interface circuitry for effectuating network connectivityvia satellite communications. Where tethered UE devices are provided aspatient devices 104, networked communications may also involve broadbandedge network infrastructures based on various flavors of DigitalSubscriber Line (DSL) architectures and/or Data Over Cable ServiceInterface Specification (DOCSIS)https://en.wikipedia.org/wiki/Help:IPA/English-compliant Cable Modem Termination System (CMTS) networkarchitectures (e.g., involving hybrid fiber-coaxial (HFC) physicalconnectivity). Accordingly, by way of illustration, an edge/accessnetwork portion 119A is exemplified with elements such as WiFi/APnode(s) 116-1, macro-cell node(s) 116-2 such as eNB nodes, gNB nodes,etc., microcell nodes 116-3 (e.g., including micro remote radio units orRRUs, etc.) and DSL/CMTS node(s) 116-4.

In similar fashion, clinicians and/or clinician agents 138 may beprovided with a variety of external devices for controlling,programming, otherwise (re)configuring or providing therapy operationswith respect to one or more patients 102 mediated via respectiveimplantable device(s) 103, in a local therapy session and/ortelehealth/remote therapy session, depending on implementation and usecase scenarios. External devices associated with clinicians/agents 138,referred to herein as clinician devices 130, which are representative ofclinician programmer device 180 shown in FIG. 1A, may comprise a varietyof UE devices, tethered or untethered, similar to patient devices 104,that may be configured to engage in telehealth and/or remote caretherapy sessions as will be set forth in detail further below. Cliniciandevices 130 may therefore also include non-COTS devices as well as COTSdevices generally exemplified by wearable device(s) 131, smartphone(s)132, tablet(s)/phablet(s) 134, computer(s) 136 and external/auxiliary AVequipment 137, any of which may operate in association with one or morevirtual assistants, smart home/office appliances, VR/AR/MR devices, andthe like. Further, example clinician devices 130 may also includevarious types of network communications circuitry or interfaces similarto that of personal devices 104, which may be configured to operate witha broad range of technologies as set forth above. Accordingly, anedge/access network portion 119B is exemplified as having elements suchas WiFi/AP node(s) 128-1, macro/microcell node(s) 128-2 and 128-3 (e.g.,including micro remote radio units or RRUs, base stations, eNB/gNBnodes, etc.) and DSL/CMTS node(s) 128-4. It should therefore beappreciated that edge/access network portions 119A, 119B may include allor any subset of wireless/wireline communication infrastructures,technologies and protocols for effectuating data communications withrespect to an example embodiment of the present disclosure.

In one arrangement, a plurality of network elements or nodes may beprovided for facilitating an integrated remote care therapy serviceinvolving one or more clinicians 138 and one or more patients 102,wherein such elements are hosted or otherwise operated by variousstakeholders in a service deployment scenario depending onimplementation, e.g., including one or more public clouds, privateclouds, or any combination thereof as previously noted. According tosome example embodiments, a remote care session management node orplatform 120 may be provided, generally representative of the networkentity 157 shown in FIG. 1A, preferably disposed as a cloud-basedelement coupled to network 118, that is operative in association with asecure communications credentials management node 122 and a devicemanagement node 124, to facilitate a virtual clinic platform whereby aclinician may advantageously engage in a telehealth session and/or aremote care therapy session with a particular patient via a commonapplication interface and associated AV and therapy controls, as will bedescribed further below.

FIG. 2 depicts a flowchart for establishing a remote programing orvirtual clinic session according to known processes. Additional detailsregarding establishment of remote programming or virtual clinic sessionsmay be found in U.S. Pat. No. 10,124,177 which is incorporated herein byreference. Although some details are described herein regardingestablishment of a virtual clinic/remote programming session, anysuitable methods may be employed according to other embodiments. Atblock 202, the patent controller device connects to the patient'smedical device. For example, the patent controller device may establisha BLUETOOTH communication session with the patient's implantable pulsegenerator. At block 204, the patient launches the patient controller appon the patient controller device. At block 206, the patient starts avirtual clinic check in process by selecting a suitable GUI component ofthe patient controller app. In block 208, the patient may providepatient credentials. At block 210, the clinician launches the clinicianprogrammer app on the clinician programmer device. In block 212, theclinician provides credentials. At block 214, the clinician checks intothe virtual clinic to communicate with the patient. At block 216, thevirtual clinic infrastructure establishes a secure connection betweenthe patient controller app and clinician programmer app to conductcommunications. Known cybersecurity features may be applied to establishthe secure connection including using PKI processes, encryptionprocesses, authentication processes, etc. Biometric data and other datamay also be employed to enhance the secure nature of the communicationsession by validating user identities and authorization. Uponestablishment, the communications may include audio and videocommunications between the patient and the clinician. Also, theclinician may conduct remote programming of the patient's medical deviceduring the session while communicating with the patient.

FIG. 3 depicts a flowchart illustrative of known blocks, steps and/oracts that may be implemented for establishing a communication sessionwith an implantable medical device. Additional details regardingestablishing a communication session with an implantable medical devicemay be found in U.S. Pat. No. 11,007,370 which is incorporated herein byreference. Although example operations are described in FIG. 3 , anysuitable methods of securing communication between a PC device and apatient medical device may be employed as appropriate for a givenpatient therapy. At block 302, a bonding procedure is initiated toestablish a trusted relationship between the implantable medical deviceor other medical device of the patient and a patient controller device.The bonding procedure may be initiated by using a magnet to activate aHall sensor in the medical device. Alternatively, near fieldcommunication (e.g., inductive coupling) may be employed to initiate thebonding procedure. The use of a magnetic or inductive coupling providesa degree of physical access control to limit the possibility ofunauthorized devices from communicating with the patient's medicaldevice. That is, a device that attempts to obtain authorization tocommunicate must be brought into physical proximity with the patient ata time that is controlled by the patient thereby reducing thepossibility of unauthorized devices from improperly gaining the abilityto communicate with the patient's device. At block 304, a communicationsession is established between the patient's medical device and thepatient's controller (PC) device. For example, a BLUETOOTH communicationsession may be established. At block 306, authentication operations areconducted. For example, known credentials, PKI, encryption, and othercybersecurity operations may be applied between the patient's medicaldevice and the PC device to determine whether the PC device should beallowed to conduct communications with the patient's medical device. Atblock 308, encryption key data may be exchanged between devices forfuture communications. At block 310, other PC identifiers or data may bestored in IMD to control future communications. Upon establishment of atrusted relationship between the patient's medica device and a PCdevice, the PC device may be used to conduct remote programming session.The remote programming sessions may also be subjected to cybersecuritymethods such as the use of credentials, PKI, encryption, etc.

FIGS. 4A and 4B depicts flowcharts illustrative of a remote carescenario involving an example digital health network architecturewherein an integrated remote care session may be established between apatient and a clinician operating respective controller devices thatsupport suitable graphical user interfaces (GUIs) for facilitating atherapy session, an audio/visual (AV) communication session, or acombination of both, for purposes of some example embodiments of thepresent disclosure. As will be set forth further below, patientcontroller and/or clinician programmer devices may be provided withappropriate application software to effectuate suitable GUIs onrespective devices for facilitating a remote care session including asecure AV session/channel and a therapy session/channel as part of acommon application interface that can support telehealth/telemedicineapplications, remote monitoring, remote therapy, remote learning, datalogging, etc. Process flow 400A of FIG. 4A may commence with a patientlaunching an integrated digital health application executing on thepatient controller/device to initiate a secure communications channelwith a remote clinician (block 402), e.g., by selecting a “Remote Care”option from a pull-down menu, clicking on an icon on the UI displayscreen, or via a voice command, etc. In one embodiment, the patient maybe ushered into a virtual waiting room, which may be realized in a UIscreen window of the patient/clinician device (block 404). At block 406,the clinician responds to the waiting patient, e.g., via a secure AVcommunication channel of the remote care session. At block 408, one ormore physiological/biological data of the patient (stored or real-time)may be provided to the clinician via secure communications. In someembodiments, one or more digital keys of the clinician and/or thepatient may be employed to secure communications. At block 410, theclinician evaluates the patient in view of the physiological/biologicaldata, telemedicine/video consultation, audio/visual cues and signalsregarding patient's facial expressions, hand movement/tremors, walking,gait, ambulatory status/stability, and other characteristics to arriveat appropriate medical assessment. Depending on such telehealthconsultation/evaluation, the clinician may remotely adjust stimulationtherapy settings for secure transmission to the patient device, whichmay be securely transmitted via encrypted communications. In a furtherscenario, a remote clinician proxy or agent may be executed at or inassociation with the patient controller/device upon launching a remotesession, wherein the proxy/agent is operative to effectuate or otherwisemediate the transmission of any therapy settings to the patient's IMD,either in real-time or at some point in the future depending uponprogrammatic control. After completing the requisite therapy andconsultative communications, the remote care session may be terminated,e.g., either by the clinician and/or the patient, as set forth at block412.

Process flow 400B of FIG. 4B is illustrative of an embodiment of a highlevel scheme for delivering healthcare to a patient via an integratedremote care session. At block 422, a remote care session between acontroller device associated with the patient and a programmer deviceassociated with a clinician may be established, wherein the clinicianand the patient are remotely located with respect to each other and theremote care session includes an AV communication session controlled byone or more AN controls provided at the patient controller device andthe clinician programmer device. At block 424, various telehealthconsultation services may be provided to the patient by the clinicianbased on interacting with the patient via the AV communication channelof the remote care session as previously noted. Responsive todetermining that the patient requires remote therapy, one or more remoteprogramming instructions may be provided to the patient's IMD via aremote therapy session or channel of the remote care session with thepatient controller device while the AV communication session ismaintained (block 426).

Skilled artisans will recognize that some of the blocks, steps and/oracts set forth above may take place at different entities and/ordifferent times (i.e., asynchronously), and possibly with interveninggaps of time and/or at different locations. Further, some of theforegoing blocks, steps and/or acts may be executed as a processinvolving just a single entity (e.g., a patient controller device, aclinician programmer device, or a remote session manager operating as avirtual clinic, etc.), or multiple entities, e.g., as a cooperativeinteraction among any combination of the end point devices and thenetwork entities. Still further, it should be appreciated that exampleprocess flows may be interleaved with one or more sub-processescomprising other IMD<=>patient or IMD<=>clinician interactions (e.g.,local therapy sessions) as well as virtual clinic<=>patient or virtualclinic<=>clinician interactions (e.g., remote patient monitoring,patient/clinician data logging, remote learning, rigidity assessment,context-aware kinematic and auditory analysis, etc., as will be setforth further below). Accordingly, skilled artisans will recognize thatexample process flows may be altered, modified, augmented or otherwisereconfigured for purposes of some embodiments herein.

In one implementation, an example remote care session may be establishedbetween the patient controller device and the clinician programmerdevice after the patient has activated a suitable GUI control providedas part of a GUI associated with the patient controller device and theclinician has activated a corresponding GUI control provided as part ofa virtual waiting room displayed on a GUI associated with the clinicianprogrammer device. In another arrangement, remote programminginstructions may be provided to the patient's IMD via the remote therapysession only after verifying that remote care therapy programming withthe patient's IMD is compliant with regulatory requirements of one ormore applicable local, regional, national, supranational governmentalbodies, non-governmental agencies, and international healthorganizations. In a still further variation, various levels of remotecontrol of a patient's controller and its hardware by a clinicianprogrammer device may be provided. For example, suitable GUI controlsmay be provided at the clinician programmer device for remotelycontrolling a camera component or an auxiliary AV device associated withthe patient controller device by interacting with a display of thepatient's image on the screen of the clinician programmer device, e.g.,by pinching, swiping, etc., to pan to and/or zoom on different parts ofthe patient in order to obtain high resolution images. Additionalembodiments and/or further details regarding some of the foregoingvariations with respect to providing remote care therapy via a virtualclinic may be found in the following U.S. patent applications,publications and/or patents: (i) U.S. Patent Application Publication No.2020/0398062, entitled “SYSTEM, METHOD AND ARCHITECTURE FOR FACILITATINGREMOTE PATIENT CARE”; (ii) U.S. Patent Application Publication No.2020/0402656, entitled “UI DESIGN FOR PATIENT AND CLINICIAN CONTROLLERDEVICES OPERATIVE IN A REMOTE CARE ARCHITECTURE”; (iii) U.S. PatentApplication Publication No. 2020/0402674, entitled “SYSTEM AND METHODFOR MODULATING THERAPY IN A REMOTE CARE ARCHITECTURE”; and (iv) U.S.Patent Application Publication No. 2020/0398063, entitled “DATA LABELINGSYSTEM AND METHOD OPERATIVE WITH PATIENT AND CLINICIAN CONTROLLERDEVICES DISPOSED IN A REMOTE CARE ARCHITECTURE”, each of which is herebyincorporated by reference herein.

FIGS. 5A and 5B depict representations of an example user interface andassociated dialog boxes or windows provided with a clinician programmerdevice, e.g., as a touch screen display of device 180 exemplified inFIG. 1A, for selecting different therapy applications and/or servicemodes in an integrated remote care service application for purposes ofsome example embodiments of the present disclosure. In one arrangement,example GUI(s) of the clinician device may be optimized or resized toprovide a maximum display window for the presentation of a patient'simage during remote therapy while allowing the presentation ofappropriate remote care therapy session and setting controls as well asAV communication session controls such that high quality video/imageinformation may be advantageously obtained by the clinician, which canhelp better evaluate the patient's response(s) to the applied/modifiedtherapy settings and/or the clinician's verbal, textual, and/or visualrequests to perform certain tasks as part of remote monitoring by theclinician. Accordingly, in some example embodiments, the cliniciandevice may be provided with one or more non-transitory tangiblecomputer-readable media or modules having program code stored thereonfor execution on the clinician device as part of or in association witha clinician programmer application for facilitating remote therapy andtelehealth delivery in an integrated session having a common applicationinterface that effectuates an optimized GUI display within the formfactor constraints of the device. In one arrangement, a code portion maybe provided for displaying a virtual waiting room identifying one ormore patients, each having at least one IMD/NIMI device configured tofacilitate a therapy, wherein the virtual waiting room is operative toaccept input by the clinician to select a patient to engage in a remotecare session with a patient controller device of the selected patient. Acode portion may be provided for displaying one or more audio controlsand one or more video controls for facilitating an AV communicationsession associated with the remote care session after the remote caresession is established between the patient controller device and theclinician programmer device. Various AV session controls may berepresented as suitable icons, pictograms, etc. as part of a GUI displayof the at the clinician programmer device, roughly similar to the GUIpresentation at a patient controller device as will be set forth below.Further, example video controls may be configured to effectuate a firstdisplay window (i.e., a clinician image window) and a second displaywindow (i.e., a patient image window) on the GUI display forrespectively presenting an image of the clinician and an image of thepatient. A code portion may be provided for displaying one or moreremote care therapy session and setting controls, wherein the one ormore remote care therapy setting controls are operative to facilitateone or more adjustments with respect to the patient's IMD settings inorder to provide appropriate therapy to the patient as part of a remotetherapy component of the remote care session. Preferably, the codeportion may be configured to provide the AV communication sessioncontrols as well as the remote care therapy session and setting controlsin a consolidated manner so as to facilitate the display thereof in aminimized overlay panel presented on the GUI screen while maximizing thesecond display window such that an enlarged presentation of thepatient's image is effectuated during the remote care session. In someembodiments, the remote care therapy setting controls may be configuredto expand into additional graphical controls for further refining one ormore IMD settings depending on the implementation and/or type(s) oftherapy applications the clinician programmer device is configured with.For example, such remote care therapy setting controls may compriseicons or pictograms corresponding to, without limitation, one or more ofa pulse amplitude setting control, a pulse width setting control, apulse frequency setting control, a pulse delay control, a pulserepetition parameter setting control, a biphasic pulse selectioncontrol, a monophasic pulse section control, a tonic stimulationselection control, a burst stimulation selection control, a leadselection control, an electrode selection control, and a “StopStimulation” control, etc., at least some of which may be presented in aset of hierarchical or nested pull-down menus or display windows. Instill further embodiments, a code portion may be provided for displayingone or more data labeling buttons as part of the GUI display of theclinician programmer device, similar to the GUI embodiments of thepatient controller device described above, wherein the one or more datalabeling buttons are operative to accept input by the cliniciancorresponding to a subjective characterization of AV quality, therapyresponse capture, and other aspects of therapy programming during theremote care session.

GUI screen 500A depicted in FIG. 5A is representative of a “login”screen that may be presented at the clinician device upon launching theclinician programmer application for facilitating a clinician to selecta service mode, e.g., a remote care service mode or an in-office careservice mode. A “Patient Room” selector menu option 502 may be operativeto present a “generator” window 505 that includes an “In-Office” patientoption 506 or a “Remote” patient option 508, wherein the activation orselection of the Remote patient option 508 effectuates one or morewindows or dialog boxes for facilitating user login, registration,authentication/authorization and other security credentialing services,as exemplified by windows 510A, 510B. Upon validation, the clinician maybe presented with a virtual waiting room 518 identifying one or moreremote patients as exemplified in GUI screen 500B of FIG. 5B. Eachremote patient may be identified by one or more identifiers and/orindicia, including, without limitation, personal identifiers, respectiveIMD identifiers, therapy identifiers, etc., subject to applicableprivacy and healthcare laws, statutes, regulations, and the like.Accordingly, in some embodiments such identification indicia maycomprise, inter alia, patient names, images, thumbnail photos, IMDserial numbers, etc., collectively referred to as Patient ID (PID)information, as illustrated by PID-1 520-1 and PID-2 520-2. In someembodiments, a time indicator may be associated with each remotepatient, indicating how long a remote patient has been “waiting” (e.g.,the time elapsed since launching a remote care session from his/hercontroller device). In some embodiments, a priority indicator may alsobe associated with remote patients, wherein different priorities may beassigned by an intervening human and/or AI/ML-based digital agent.Furthermore, patients may have different types of IMDs to effectuatedifferent therapies and a patient may have more than one IMD in somecases. An example embodiment of virtual waiting room 518 may thereforeinclude a display of any combination of remote patients and theirrespective IMDs by way of suitably distinguishable PIDs having variouspieces of information, wherein the PIDs may be individually selectableby the clinician for establishing a remote care session that may includeremote therapy programming or just telehealth consultations.

FIG. 6 depicts a representation of an example user interface of aclinician programmer device with additional details for facilitatinggraphic controls with respect to an AV communication session and aremote therapy session in an integrated remote care service applicationfor purposes of some embodiments of the present disclosure. Asillustrated, GUI screen 600 is representative of a display screen thatmay be presented at the clinician device after establishing that remotetherapy programming is to be effectuated for a selected remote patient.In accordance with some of the embodiments set forth herein, GUI screen600 may be arranged so that the patient's video image is presented in anoptimized or resized/oversized display window 602 while the clinician'svideo image is presented in a smaller display window 604 along with acompact control icon panel 606 to maximize the level ofdetail/resolution obtained in the patient's image. Furthermore, thesmaller clinician image window 604 may be moved around the UI screen by“dragging” the image around the viewing area of the patient's imagewindow 602 to allow more control of the positioning of the displaywindows so that the patient's image view is unimpeded and/or optimizedat a highest possible resolution. It will be appreciated that such highlevel video quality is particularly advantageous in obtaining morereliable cues with respect to the patient's facial expressions, moods,gestures, eye/iris movements, lip movements, hand movements tremors,jerks, twitches, spasms, contractions, or gait, etc., that may be usefulin diagnosing various types of motor/neurological disorders, e.g.,Parkinson's disease. As will be seen further below, a remote datalogging platform may be configured to store the AV data of the sessionsfor facilitating model building and training by appropriate AI/ML-basedexpert systems or digital assistants for purposes of further embodimentsof the present patent disclosure.

Control panel window 606 may include a sub-panel of icons for AV and/orremote care session controls, e.g., as exemplified by sub-panel 607 inaddition to a plurality of icons representing remote therapy settingcontrols, e.g., pulse amplitude control 608, pulse width control 610,pulse frequency control 612, increment/decrement control 614 that may beused in conjunction with one or more therapy setting controls, alongwith a lead selection indication icon 619. Skilled artisans willrecognize that the exact manner in which a control panel window may bearranged as part of a consolidated GUI display depends on the therapyapplication, IMD deployment (e.g., the number of leads, electrodes perlead, electrode configuration, etc.), and the like, as well as theparticular therapy settings. Additional control icons relating tostimulation session control, e.g., Stop Stimulation icon 609, as well asany other icons relating to the remote care session such aslead/electrode selection 613, may be presented as minimized sub-panelsadjacent to the control panel window 606 so as not to compromise thedisplay area associated with the patient' image display 602.

In some embodiments, a code portion may be provided as part of theclinician programmer application to effectuate the transitioning of GUIscreen 600 to or from a different sizing (e.g., resizing) in order tofacilitate more expanded, icon-rich GUI screen in a different displaymode. For example, a client device GUI screen may be configured suchthat the clinician's and patient's video images are presented in smallerwindows, respectively, with most of the rest of the display region beingpopulated by various icons, windows, pull-down menus, dialog boxes,etc., for presenting available programming options, lead selectionoptions, therapy setting options, electrode selection options, and thelike, in a more elaborate manner. In some embodiments, the video UIpanels and related controls associated with clinician/patient videoimage windows may be moved around the GUI screen by “dragging” theimages around the display area. Still further, the positioning of thevideo UI panels and related controls associated with clinician/patientvideo image windows may be stored as a user preference for a future UIsetup or configuration that can be instantiated or initialized when thecontroller application is launched. As can be appreciated, it iscontemplated that a clinician device may be configured to be able totoggle between multiple GUI display modes by pressing or otherwiseactivating zoom/collapse buttons that may be provided on respectivescreens.

In some further embodiments, a clinician device may be provided withadditional functionality when utilizing or operating in the resizeddisplay GUI screen mode. By way of a suitable inputting mechanism at theclinician device, e.g., by pressing or double-tapping a particularportion of the patient's image, or by scrolling a cursor or a pointingdevice to a particular potion of the patient's image, etc., theclinician can remotely control the AV functionality of the patientcontroller device, e.g., a built-in camera or an auxiliary AV devicesuch as AV equipment, in order to zoom in on and/or pan to specificportions of the patient's body in order to obtain close-up images thatcan enable better diagnostic assessment by the clinician. In suchembodiments, zooming or enlarging of a portion of the patient's image,e.g., eye portion, may be effectuated by either actual zooming, i.e.,physical/optical zooming of the camera hardware, or by way of digitalzooming (i.e., by way of image processing).

In some embodiments, both optical and digital zooming of a patient'simage may be employed. In still further embodiments, the patientcontroller device and/or associated AV equipment may be panned and/ortilted to different portions of the patient's body to observe variousmotor responses and/or conditions while different programming settingsmay be effectuated in a remote therapy session, e.g., shaking andtremors, slowed movement or bradykinesia, balance difficulties andeventual problems standing up, stiffness in limbs, shuffling whenwalking, dragging one or both feet when walking, having little or nofacial expressions, drooling, muscle freezing, difficulty with tasksthat are repetitive in nature (like tapping fingers or clapping hands orwriting), difficulty in performing everyday activities like buttoningclothes, brushing teeth, styling hair, etc.

In still further embodiments, separate remote therapy sessionintervention controls (e.g., pause and resume controls) may be providedin addition to stimulation start and termination controls, which may beoperative independent of or in conjunction with AV communication sessioncontrols, in a manner similar to example patient controller GUIembodiments set forth hereinbelow. Still further, data labeling buttonsor controls may also be provided in a separate overlay or window of GUIscreen 600 (not shown in FIG. 6 ) to allow or otherwise enable theclinician to provide different types of data labels for the AV data andtherapy settings data for purposes of some embodiments of the presentpatent disclosure.

FIG. 7 depicts a block diagram of a generalized external edge deviceoperative in a digital health network architecture for purposes of someembodiments of the present disclosure. For example, depending onconfiguration and/or modality, external device 700 may be representativeof a clinician programmer device, a patient controller device, or adelegated device operated by an agent of a patient or a clinician havingsubordinate levels of privilege authorization with respect to remotetherapy and monitoring operations. Further, external device 700 may be aCOTS device or non-COTS device as previously noted. Still further,external device 700 may be a device that is controlled and managed in acentralized enterprise device management system (EDMS), also referred toas a mobile/medical device management system (MDMS), which may beassociated with the manufacturer of the IMDs and associated therapyapplication components in some embodiments (e.g., as an intranetimplementation, an extranet implementation, or internet-based cloudimplementation, etc.), in order to ensure that only appropriatelymanaged/provisioned devices and users are allowed to engage incommunications with IMDs with respect to monitoring the devices and/orproviding therapy to patients using approved therapy applications. Stillfurther, external device 700 may be a device that is not controlled andmanaged in such a device management system. Accordingly, it will berealized that external device 700 may comprise a device that may beconfigured in a variety of ways depending on how its functional modalityis implemented in a particular deployment.

Example external device 700 may include one or more processors 702,communication circuitry 718 and one or more memory modules 710,operative in association with one or more OS platforms 704 and one ormore software applications 708-1 to 708-K depending on configuration,cumulatively referred to as software environment 706, and any otherhardware/software/firmware modules, all being powered by a power supply722, e.g., battery. Example software environment 706 and/or memory 710may include one or more persistent memory modules comprising programcode or instructions for controlling overall operations of the device,inter alia. Example OS platforms may include embedded real-time OSsystems, and may be selected from, without limitation, iOS, Android,Chrome OS, Blackberry OS, Fire OS, Ubuntu, Sailfish OS, Windows, Kai OS,eCos, LynxOS, QNX, RTLinux, Symbian OS, VxWorks, Windows CE, MontaVistaLinux, and the like. In some embodiments, at least a portion of thesoftware applications may include code or program instructions operativeas one or more medical/digital health applications for effectuating orfacilitating one or more therapy applications, remote monitoring/testingoperations, data capture and logging operations, trial therapyapplications, etc. Such applications may be provided as a singleintegrated app having various modules that may be selected and executedvia suitable drop-down menus in some embodiments. However, variousaspects of the edge device digital healthcare functionalities may alsobe provided as individual apps that may be downloaded from one or moresources such as device manufactures, third-party developers, etc. By wayof illustration, application 708-1 is exemplified as digital healthcareapp configured to interoperate with program code stored in memory 710 toexecute various operations relative to device registration, modeselection, remote/test/trial programming, therapy selection, securityapplications, and provisioning, etc., as part of a device controllerapplication.

In some embodiments of external device 700, memory modules 710 mayinclude a non-volatile storage area or module configured to storerelevant patient data, therapy settings, and the like. Memory modules710 may further include a secure storage area 712 to store a deviceidentifier (e.g., a serial number) of device 700 used during therapysessions (e.g., local therapy programming or remote therapyprogramming). Also, memory modules 710 may include a secure storage area714 for storing security credential information, e.g., one or morecryptographic keys or key pairs, signed digital certificates, etc. Insome arrangements, such security credential information may bespecifically operative in association with approved/provisioned softwareapplications, e.g., therapy/test application 708-1, which may beobtained during provisioning. Also, a non-volatile storage area 716 maybe provided for storing provisioning data, validation data, settingsdata, metadata etc. Communication circuitry 718 may include appropriatehardware, software and interfaces to facilitate wireless and/or wirelinecommunications, e.g., inductive communications, wireless telemetry orM2M communications, etc. to effectuate IMD communications, as well asnetworked communications with cellular telephony networks, local areanetworks (LANs), wide area networks (WANs), packet-switched datanetworks, etc., based on a variety of access technologies andcommunication protocols, which may be controlled by the digitalhealthcare application 708-1 depending on implementation.

For example, application 708-1 may include code or program instructionsconfigured to effectuate wireless telemetry and authentication with anIMD/NIMI device using a suitable M2M communication protocol stack whichmay be mediated via virtual/digital assistant technologies in somearrangements. By way of illustration, one or more bi-directionalcommunication links with a device may be effectuated via a wirelesspersonal area network (WPAN) using a standard wireless protocol such asBluetooth Low Energy (BLE), Bluetooth, Wireless USB, Zigbee, Near-FieldCommunications (NFC), WiFi (e.g., IEEE 802.11 suite of protocols),Infrared Wireless, and the like. In some arrangements, bi-directionalcommunication links may also be established using magnetic inductiontechniques rather than radio waves, e.g., via an induction wirelessmechanism. Alternatively and/or additionally, communication links may beeffectuated in accordance with certain healthcare-specificcommunications services including, Medical Implant Communication Service(MICS), Wireless Medical Telemetry Service (MTS), Medical DeviceRadiocommunications Service (MDRS), Medical Data Service (MDS), etc.Accordingly, regardless of which type(s) of communication technologybeing used, external device 700 may be provided with one or morecommunication protocol stacks 744 operative with hardware, software andfirmware (e.g., forming suitable communication circuitry includingtransceiver circuitry and antenna circuitry where necessary, which maybe collectively exemplified as communication circuitry 718 as previouslynoted) for effectuating appropriate short-range and long-rangecommunication links for purposes of some example embodiments herein.

External device 700 may also include appropriate audio/video controls720 as well as suitable display(s) (e.g., touch screen), camera(s),microphone, and other user interfaces (e.g., GUIs) 742, which may beutilized for purposes of some example embodiments of the presentdisclosure, e.g., facilitating user input, initiating IMD/networkcommunications, mode selection, therapy selection, etc., which maydepend on the aspect(s) of a particular digital healthcare applicationbeing implemented.

FIG. 8 depicts a block diagram illustrating additional detailspertaining to a patient controller device operative in a digital healthnetwork architecture for purposes of some embodiments of the presentdisclosure. Example patient controller device 800 may be particularlyconfigured for securely packaging and transmitting patient data to anexternal entity, e.g., a clinician programmer device and/or a networkentity disposed in the digital health network in order to facilitateremote monitoring, AI/ML model training, and the like. Consistent withthe description provided above with respect to a generalized edgedevice, patient controller device 800 may be provided with a patientcontroller application 802 configured to run in association with asuitable device hardware/software environment 850 effectuated by one ormore processor and memory modules 806, one or more OS platforms 808, andone or more persistent memory modules 816 comprising program code orinstructions for controlling overall operations of the device, interalia. Example OS platforms may include a variety of embedded real-timeOS systems as noted previously. In one implementation, a secure filesystem 810 that can only be accessed by the patient controllerapplication 802 may be provided, wherein one or more patient data files812 may be stored in a packaged encrypted form for secure transmissionfor purposes of some embodiments herein. Also, patient controllerapplication 802 may include a therapy manager 824 operative tofacilitate remote and/or non-remote therapy applications and relatedcommunications using one or more communication interfaces, e.g.,interface 834 with an IPG/IMD 804 and network communications interface836 with a network entity, as previously noted. A logging manager 830associated with therapy manager 824 may be provided for logging data. Asecurity manager 828 associated with therapy manager 824 may be providedfor facilitating secure or trusted communications with a network entityin some embodiments. A therapy communication manager 832 may be providedfor facilitating secure therapy communications between patientcontroller 800 and a clinician programmer (not shown in this FIG.).Therapy communication manager 832 may also be interfaced with localcommunication interface 834 to effectuate secure communications with thepatient's IPG/IMD 804 using a suitable short-range communicationstechnology or protocol as noted previously.

In still further arrangements, suitable software/firmware modules 820may be provided as part of patient controller application 802 toeffectuate appropriate user interfaces and controls, e.g., NV GUIs, inassociation with an audio/video manager 822 for facilitatingtherapy/diagnostics control, file management, and/or other input/output(I/O) functions. Additionally, patient controller 800 may include anencryption module 814 operative independently and/or in association orotherwise integrated with patient controller application 802 fordynamically encrypting a patient data file, e.g., on a line-by-linebasis during runtime, using any known or heretofore unknown symmetricand/or asymmetric cryptography schemes, such as the Advanced EncryptionStandard (AES) scheme, the Rivest-Shamir-Adleman (RSA) scheme, EllipticCurve Cryptography (ECC), etc.

FIG. 9 depicts a block diagram illustrating additional detailspertaining to a clinician programmer device operative in a digitalhealth network architecture for purposes of some embodiments of thepresent disclosure. Similar to the example patient controller device 800described above, example clinician programmer 900 may be particularlyconfigured for facilitating secure transmission of patient data to anexternal entity, e.g., another clinician programmer device and/or anetwork entity disposed in the digital healthcare network in order tofacilitate remote monitoring, AI/ML model training, and the like. Aclinician programmer application 902 may be configured to run inassociation with a suitable device hardware/software environment 950effectuated by one or more processor and memory modules 304, one or moreOS platforms 906, and one or more persistent memory modules 914comprising program code or instructions for controlling overalloperations of the device, inter alia. As before, example OS platformsmay include a variety of embedded real-time OS systems according to someembodiments. Further, a secure file system 908 may be provided inclinician programmer 900 that can only be accessed by the clinicianprogrammer application 902, wherein one or more patient data files 310(e.g., corresponding to one or more patients) may be stored in apackaged encrypted form, respectively, for purposes of some embodimentsherein. In one implementation, clinician programmer application 902 mayinclude a therapy manager 926 operative to facilitate remote and/ornon-remote therapy applications and related communications using one ormore communication interfaces, e.g., interface 924. For example,interface 924 may be configured to communicate with an IMD (not shown inthis FIG.) using various short-range communication links with respect toin-person or in-clinic therapy according to some embodiments aspreviously noted. Likewise, example interface 924 may be configured toprovide connectivity with wide-area networks for facilitating remoteprogramming of an IMD and/or a telehealth session in some scenarios. Alogging manager 928 associated with therapy manager 924 may be providedfor logging data for respective patients. A security manager 930associated with therapy manager 926 may be provided for facilitatingsecure or trusted communications with a network entity in someembodiments. A therapy communication manager 932 may be provided forfacilitating secure therapy communications between clinician programmer900 and a patient controller (not shown in this FIG.). Suitablesoftware/firmware modules 920 may be provided as part of clinicianprogrammer application 902 to effectuate appropriate user interfaces andcontrols, e.g., NV GUIs, in association with an audio/video manager 922for facilitating therapy/diagnostics control, file management, and/orother I/O functions as noted previously. Further, clinician programmer900 may include an encryption module 912 similar to that of patientcontroller 800, wherein the encryption module 912 is operative inassociation and/or otherwise integrated with clinician programmerapplication 902 for encrypting a patient data file, e.g., dynamically ona line-by-line basis, during runtime using suitable techniques.

FIG. 10 depicts a block diagram of an IMD and associated system that maybe configured for facilitating a remote care therapy application and/ora local therapy session for purposes of an example embodiment of thepresent disclosure. In general, therapy system 1000 may be adapted togenerate electrical pulses to stimulate spinal cord tissue, peripheralnerve tissue, deep brain tissue, DRG tissue, cortical tissue, cardiactissue, digestive tissue, pelvic floor tissue, or any other suitablebiological tissue of interest within a patient's body, using an IMD or atrial IMD as previously noted. In one example embodiment, IMD 1002 maybe implemented as having a metallic housing or can that encloses acontroller/processing block or module 1012, pulse generating circuitryincluding or associated with one or more stimulation engines 1010, acharging coil 1016, a power supply or battery 1018, a far-field and/ornear field communication block or module 1024, battery chargingcircuitry 1022, switching circuitry 1020, sensing circuitry 1026, amemory module 1014, and the like. IMD 1002 may include a diagnosticcircuit module associated with a sensing module 1026 adapted toeffectuate various diagnostics with respect to the state/condition ofone or more stimulation electrodes and sensing electrodes of animplantable lead system as well as other bio/physiological sensorsintegrated or otherwise operative with IMD 1002. Controller/processormodule 1012 typically includes a microcontroller or other suitableprocessor for controlling the various other components of IMD 1302.Software/firmware code, including digital healthcare application andencryption functionality, may be stored in memory 1014 of IMD 1002,and/or may be integrated with controller/processor module 1012. Otherapplication-specific software code as well as associated storagecomponents (not particularly shown in this FIG.) for execution by themicrocontroller or processor 1012 and/or other programmable logic blocksmay be provided to control the various components of the device forpurposes of an embodiment of the present patent disclosure. As such,example IMD 1002 may be adapted to generate stimulation pulses accordingto known or heretofore known stimulation settings, programs, etc.

In one arrangement, IMD 1002 may be coupled (via a “header” as is knownin the art, not shown in this FIG.) to a lead system having a leadconnector 1008 for coupling a first component 1006A emanating from IMD1002 with a second component 1006B that includes a plurality ofelectrodes 1004-1 to 1004-N, which may be positioned proximate to thepatient tissue. Although a single lead system 1006A/1006B isexemplified, it should be appreciated that an example lead system mayinclude more than one lead, each having a respective number ofelectrodes for providing therapy according to configurable settings. Forexample, a therapy program may include one or more lead/electrodeselection settings, one or more sets of stimulation parameterscorresponding to different lead/electrode combinations, respectively,such as pulse amplitude, stimulation level, pulse width, pulse frequencyor inter-pulse period, pulse repetition parameter (e.g., number of timesfor a given pulse to be repeated for respective stimulation sets or“stimsets” during the execution of a program), etc. Additional therapysettings data may comprise electrode configuration data for delivery ofelectrical pulses (e.g., as cathodic nodes, anodic nodes, or configuredas inactive nodes, etc.), stimulation pattern identification (e.g.,tonic stimulation, burst stimulation, noise stimulation, biphasicstimulation, monophasic stimulation, and/or the like), etc. Stillfurther, therapy programming data may be accompanied with respectivemetadata and/or any other relevant data or indicia.

As noted previously, external device 1030 may be deployed for use withIMD 1002 for therapy application, management and monitoring purposes,e.g., either as a patient controller device or a clinician programmerdevice. In general, electrical pulses are generated by the pulsegenerating circuitry 1010 under the control of processing block 1012,and are provided to the switching circuitry 1020 that is operative toselectively connect to electrical outputs of IMD 1002, wherein one ormore stimulation electrodes 1004-1 to 1004-N per each lead 1006A/B maybe energized according to a therapy protocol, e.g., by the patient orpatient's agent (via a local session) and/or a clinician (via a local orremote session) using corresponding external device 1030. Also, externaldevice 1030 may be implemented to charge/recharge the battery 1018 ofIPG/IMD 1002 (although a separate recharging device could alternativelybe employed), to access memory 1012/1014, and/or to program or reprogramIMD 1002 with respect to one or more stimulation set parametersincluding pulsing specifications while implanted within the patient. Inalternative embodiments, however, separate programmer devices may beemployed for charging and/or programming the IMD device 1002 deviceand/or any programmable components thereof. Software stored within anon-transitory memory of the external device 1030 may be executed by aprocessor to control the various operations of the external device 1030,including facilitating encryption of patient data logged in or by IMD1002 and extracted therefrom. A connector or “wand” 1034 may beelectrically coupled to the external device 430 through suitableelectrical connectors (not specifically shown), which may beelectrically connected to a telemetry component 1032 (e.g., inductorcoil, RF transceiver, etc.) at the distal end of wand 1034 throughrespective communication links that allow bi-directional communicationwith IMD 1002. Alternatively, there may be no separate or additionalexternal communication/telemetry components provided with externaldevice 1030 in an example embodiment that uses BLE or the like forfacilitating bi-directional communications with IMD 1002.

In a setting involving in-clinic or in-person operations, a user (e.g.,a doctor, a medical technician, or the patient) may initiatecommunication with IMD 1002. External device 1030 preferably providesone or more user interfaces 1036 (e.g., touch screen, keyboard, mouse,buttons, scroll wheels or rollers, or the like), allowing the user tooperate IMD 1002. External device 1030 may be controlled by the userthrough user interface 1036, allowing the user to interact with IMD1002, whereby operations involving therapy application/programming,coordination of patient data security including encryption, trial IMDdata report processing, etc., may be effectuated.

FIGS. 11A-11H depict representations of an example user interface andassociated dialog boxes or windows provided with a patient controllerdevice for selecting different therapy applications and/or service modesand for facilitating controls with respect to an AV communicationsession as well as a remote therapy session in an integrated remote careservice application for purposes of an embodiment of the presentdisclosure. In some example implementations, a patient controllerdevice, e.g. device 150 shown in FIG. 1A, may be provided with one ormore non-transitory tangible computer-readable media or modules havingprogram code stored thereon for execution on the patient controllerdevice as part of or in association with a patient controllerapplication, e.g., application 152, for facilitating remote therapy andtelehealth applications in an integrated session having a commonapplication interface. A code portion may be provided for displaying amode selector icon on a GUI display screen of the patient controllerdevice, wherein the mode selector icon is operative for accepting inputby the patient to launch a remote care session with a clinician having aclinician programmer device. A code portion may be provided fordisplaying one or more audio controls and one or more video controls forfacilitating an AV communication session or channel associated with theremote care session after the remote care session is established betweenthe patient controller device and the clinician programmer device. SuchAV controls may be represented as suitable icons, pictograms, and thelike, e.g., a video/camera icon for controlling a video channel, amicrophone icon for controlling an audio channel, a speaker icon forvolume control, as well as control icons operative with respect topicture-in-picture (PIP) display regions, and the like. For example,video controls may be operative to effectuate a first display window anda second display window on the GUI display for respectively presentingan image of the clinician and an image of the patient in a PIP displaymode. Yet another code portion may be provided for displaying one ormore remote care therapy session controls in an overlay panel presentedon the GUI display, wherein the one or more remote care therapy sessioncontrols are operative with respect to starting and ending a remote caretherapy session by the patient as well as facilitating a temporaryintervention or interruption of the therapy session while the AVcommunication session is maintained. As noted above, an example remotecare therapy session may involve providing one or more programminginstructions to the patient's IMD as part of the remote care session,and temporary intervention of the remote therapy may only suspend theremote programming of the patient's IMD although the AV communicationsession between the patient and the clinician remains active. In furtherembodiments, one or more code portions may be provided with the patientcontroller application to effectuate tactile controls with respect todifferent portions, fields, regions or X-Y coordinates of an active GUIdisplay window that may be configured to interact with the functionalityof the AV controls and/or therapy session controls. In still furtherembodiments, one or more code portions may be provided with the patientcontroller application to effectuate one or more data labeling buttons,icons, pictograms, etc., as part of the GUI display of the patientcontroller device, wherein the one or more data labeling buttons areoperative to accept input by the patient corresponding to a subjectivecharacterization of audio and/or video quality of the AV communicationsand/or other aspects of the therapy by the patient during the remotecare session. In still further embodiments, one or more code portionsmay be provided with the patient controller application to facilitatepatient input/feedback with respect to a trial therapy or treatmentinvolving an IMD or a NIMI device, which may be augmented with one ormore data labeling buttons, icons, pictograms, etc., wherein the patientinput/feedback data may be provided to a network-based AI/ML model forfacilitating intelligent decision-making with respect to whether theIMD/NIMI device should be deployed in a more permanent manner (e.g.,implantation) and/or whether a particular therapy setting or a set ofsettings, including context-sensitive therapy program selection, mayneed to be optimized or otherwise reconfigured.

As illustrated, FIGS. 11A and 11B depict example GUI screens 1100A and1100B of a patient controller device that allow user input with respectto various mode settings/selections, including the activation anddeactivation of allowing a remote control programming (i.e., therapy)session to be conducted, e. g., in trial therapy mode, etc. GUI displayscreen 1100A includes a mode selector 1102 that may be activated to showvarious mode settings which in turn may be selected, enabled orotherwise activated by using associated tactile controls. For example,modes such as “Airplane Ready” 1104A, “Surgery Mode” 1106A, “MRI Mode”1108A, “Remote Control Mode” 1110A, “Trial Mode” 1112A, each having acorresponding swipe button 1104B-1112B are depicted. GUI screen 1100Billustrates a display that may be effectuated upon selecting or allowingRemote Control 1120 wherein a Remote Care Mode 1122A may be selected orenabled for activating remote therapy using a corresponding swipe button1122B. A patient may therefore selectively permit the activation ofremote therapy (i.e., remote programming of the IMD), whereby ifactivated and connected, a clinician can securely change or modify thetherapy settings of the patient's IMD by effectuating appropriatetherapy setting controls and associated GUIs provided at a controllerdevice as previously set forth.

FIG. 11C depicts an example GUI display screen 1100C of the patientcontroller device during a remote care session, wherein an image of theselected clinician 1142 and an image of the patient 1144 may bepresented in a PIP display region. In one display mode, the patient'simage 1144 may be presented as a smaller offset or overlay image and theclinician's image 1142 may be presented as a main, larger image. In someembodiments, the patient image window 1144 may be moved around the UIscreen by “dragging” the image around the viewing window allocated tothe clinician image 1142. An image swap control 1146 may be provided toswap the PIP display regions in another display mode, whereby thepatient's image 1144 may be presented as the main, larger image whereasthe clinician's image 1142 may be presented in a smaller overlay window.

In some embodiments, a control panel 1140 may also be presented as partof the GUI screen 1100C, wherein various AV communication sessioncontrols and remote therapy session controls may be displayed assuitable icons, pictograms, etc., in a consolidated GUI display as notedabove. A video session icon 1130 may be activated/enabled ordeactivated/disabled to selectively turn on or off the video channel ofthe session. A microphone icon 1134 may be activated/enabled ordeactivated/disabled to selectively turn on or off the audio channel ofthe session. A pause/resume icon 1132 may be activated/enabled ordeactivated/disabled to selectively pause or suspend, or resume theremote therapy session involving remote programming of the patient's IMDor any other remote digital healthcare application executing on thepatient controller. In some implementations, activating or deactivatingthe video session icon 1130 may also be configured to turn on or off theremote therapy session. In some implementations, separate remote therapysession controls (e.g., start control, end control, etc. in addition topause and resume controls) may be provided that are operativeindependent of the AV communication session controls. Still further,data labeling buttons may also be provided in a separate overlay orwindow of the GUI screen 1100C (not shown in this FIG.) to allow orotherwise enable the patent to input a subjective characterization ofthe AV data and therapy experience data as noted previously.

In a further embodiment of a digital health network architecture of thepresent patent disclosure, a digital health “app” may be installed on ordownloaded to a patient controller device, e.g., patient controllerdevice 1210 shown in FIG. 12 , to permit a patient to report therapyoutcomes for clinician review and analysis. An example of an existingdigital health app that is available to patients in certainjurisdictions for reporting therapy outcomes to neurostimulation(including spinal cord stimulation) is the MYPATH TM app from Abbott(Plano, Tex.). The digital health app may use the network communicationscapabilities of patient controller device 1210 to communicatepatient-reported data to patient report processing platform 1218. Theclinician of a given patient may review the patient-reported data storedon platform 1218 to determine whether the therapy is working as expectedand whether the patient requires reprogramming to optimize therapy. Insome cases, the patient may provide the patient-reported data during a“trial” period which is used to evaluate the effectiveness of thetherapy for the patient from a temporary external system before surgicalimplantation of the IMD. If the trial is successful, surgicalimplantation of the IMD may occur. Additionally or alternatively, thepatient may provide the outcome data after surgical implantation of theIMD to allow monitoring of the patient's condition and response to thetherapy to continue on an ongoing basis.

Turning to FIG. 11D, depicted therein is an example GUI display screen1100D of a patient controller device for facilitating reporting ofpatient outcomes (either during an initial trial period to evaluatetherapy or during long-term use of the therapy). The GUI display forreporting patient outcomes may be provided as a single app or may beintegrated with other digital health apps (such as a remoteprogramming/virtual clinic app and/or the patient controller app forcontrolling operations of the IMD or minimally invasive device). ExampleGUI display screen 1100F shown in FIG. 11F is illustrative of a menupanel for use by the patient to input data related to the patient'scondition related to the therapy provided by the patient's IMD orminimally invasive device. By way of example, a Pain Location Selectionmenu 1160, a Baseline Pain Level menu 1162, a Baseline Well-being menu1164, a Baseline Survey menu 1166 and a Preferences and Reminders menu1168 are illustrated. In one arrangement, Pain Location Selection menu1160 is operative to enable the patient to select/identify one or morelocations on the patient's body where pain is located or perceived,e.g., left foot, right foot, ankle(s), knee(s), leg(s), pelvis, groin,hip, abdomen, upper/lower back, hand(s), arm(s), etc.

In some example arrangements, baseline data regarding pain levels (e.g.,as a whole and/or for identified bodily regions), sense of well-being,measurements of physiologic and behavioral markers may be establishedfor the patients, wherein each patient may select a varying trialperiod, e.g., each day, each week, 2 weeks, etc. Patients may answer aplurality of questions with respect to each baseline, wherein theanswers may be alphanumeric input (e.g., on a scale of 0 to 10), graphicinput, or A/V input, or any combination thereof (as shown in GUI 1100Eand GUI 1100H in FIGS. 11E and 11H respectively as examples). One ormore questionnaires 1170, 1172 may be provided as part of a GUI displayscreen 1100G for purposes of obtaining patient input(s), as exemplifiedin FIG. 11G, at least some of which may be presented in a set ofhierarchical or nested pull-down menus or dialog boxes.

In some example arrangements, various pieces of data and informationfrom the end points disposed in a digital healthcare networkarchitecture, e.g., architecture 1260 shown in FIG. 12 , may also betransmitted to one or more cloud-centric platforms without end userinvolvement, e.g., as background data collection processes, in additionto user-initiated secure data transfer operations.

As previously noted, one or more remote data logging platforms 1216 ofsystem 1200 (shown in FIG. 12 ) may be configured to obtain, receive orotherwise retrieve data from patient controller devices, clinicianprogrammer devices and other authorized third-party devices. On anindividual patient level and on a patient population basis, patientaggregate data 1250 is available for processing, analysis, and review tooptimize patient outcomes for individual patients, for a patientpopulation as a whole, and for relevant patient sub-populations ofpatients.

Patient aggregate data (PAD) 1250 may include basic patient dataincluding patient name, age, and demographic information, etc. PAD 1250may also include information typically contained in a patient's medicalfile such as medical history, diagnosis, results from medical testing,medical images, etc. The data may be inputted directly into system 1200by a clinician or medical professional. Alternatively, this data may beimported from digital health records of patients from one or more healthcare providers or institutions.

As previously discussed, a patient may employ a patient controller “app”on the patient's smartphone or other electronic device to control theoperations of the patient's IMD or minimally invasive device. Forexample, for spinal cord stimulation or dorsal root stimulation, thepatient may use the patient controller app to turn the therapy on andoff, switch between therapy programs, and/or adjust stimulationamplitude, frequency, pulse width, and/or duty cycle, among otheroperations. The patient controller app is adapted to log such events(“Device Use/Events Data”) and communicate the events to system 1200 tomaintain a therapy history for the patient for review by the patient'sclinician(s) to evaluate and/or optimize the patient's therapy asappropriate.

PAD 1250 may include “Patient Self-Report Data” obtained using a digitalhealth care app operating on patient controller devices 1210. Thepatient self-report data may include patient reported levels of pain,patient well-being scores, emotional states, activity levels, and/or anyother relevant patient reported information. The data may be obtainedusing the MYPATH app from Abbott as one example.

PAD 1250 may include sensor data. For example, IMDs of patients mayinclude integrated sensors that sense or detect physiological activityor other patient states. Example sensor data from IMDs may include datedrelated to evoked compound action potentials (ECAPs), local fieldpotentials, EEG activity, patient heart rate or other cardiac activity,patient respiratory activity, metabolic activity, blood glucose levels,and/or any other suitable physiological activity. The integrated sensorsmay include position sensing circuits and/or accelerometers to monitorphysical activity of the patient. Data captured using such sensors canbe communicated from the medical devices to patient controller devicesand then stored within patient/clinician data logging and monitoringplatform 1216. Patients may also possess wearable devices (see, e.g.,device 106 in FIG. 1B) such as health monitoring products (heart ratemonitors, fitness tracking devices, smartwatches, etc.). Any dataavailable from wearable devices may be likewise communicated tomonitoring platform 1216.

As previously discussed, patients may interact with clinicians usingremote programming/virtual clinic capabilities of system 1200. The videodata captured during virtual clinic and/or remote programming sessionsmay be archived by platform 1214. The video from these sessions may besubjected to automated video analysis (contemporaneously with thesessions or afterwards) to extract relevant patient metrics. PAD data1250 may include video analytic data for individual patients, patientsub-populations, and the overall patient population for each supportedtherapy.

The data may comprise various data logs that capture patient-clinicianinteractions (“Remote Programming Event Data” in PAD 1250), e.g.,individual patients' therapy/program settings data in virtual clinicand/or in-clinic settings, patients' interactions with remote learningresources, physiological/behavioral data, daily activity data, and thelike. Clinicians may include clinician reported information such aspatient evaluations, diagnoses, etc. in PAD 1250 via platform 1216 insome embodiments. Depending on implementation, the data may betransmitted to the network entities via push mechanisms, pullmechanisms, hybrid push/pull mechanisms, event-driven or trigger-baseddata transfer operations, and the like.

In some example arrangements, data obtained via remote monitoring,background process(es), baseline queries and/or user-initiated datatransfer mechanisms may be (pre)processed or otherwise conditioned inorder to generate appropriate datasets that may be used for training,validating and testing one or more AI/ML-based models or engines forpurposes of some embodiments. In some example embodiments, patient inputdata may be securely transmitted to the cloud-centric digital healthcareinfrastructure wherein appropriate AI/ML-based modeling techniques maybe executed for evaluating the progress of the therapy trial, predictingefficacy outcomes, providing/recommending updated settings, etc.

In one implementation, “Big Data” analytics may be employed as part of adata analytics platform, e.g., platform 1220, of a cloud-centric digitalhealth infrastructure 1212. In the context of an example implementationof the digital health infrastructure 1212, “Big Data” may be used as aterm for a collection of datasets so large and complex that it becomesvirtually impossible to process using conventional database managementtools or traditional data processing applications. Challenges involving“Big Data” may include capture, curation, storage, search, sharing,transfer, analysis, and visualization, etc. Because “Big Data” availablewith respect to patients' health data, physiological/behavioral data,sensor data gathered from patients and respective ambient surroundings,daily activity data, therapy settings data, health data collected fromclinicians, etc. can be on the order of several terabytes to petabytesto exabytes or more, it becomes exceedingly difficult to work with usingmost relational database management systems for optimizing, ranking andindexing search results in typical environments. Accordingly, exampleAI/ML processes may be implemented in a “massively parallel processing”(MPP) architecture with software running on tens, hundreds, or eventhousands of servers. It should be understood that what is considered“Big Data” may vary depending on the capabilities of the datacenterorganization or service provider managing the databases, and on thecapabilities of the applications that are traditionally used to processand analyze the dataset(s) for optimizing ML model reliability. In oneexample implementation, databases may be implemented in an open-sourcesoftware framework such as, e.g., Apache Hadoop, that is optimized forstorage and large-scale processing of datasets on clusters of commodityhardware. In a Hadoop-based implementation, the software framework maycomprise a common set of libraries and utilities needed by othermodules, a distributed file system (DFS) that stores data on commoditymachines configured to provide a high aggregate bandwidth across thecluster, a resource-management platform responsible for managing computeresources in the clusters and using them for scheduling of AI/ML modelexecution, and a MapReduce-based programming model for large scale dataprocessing.

In one implementation, data analytics platform 1220 may be configured toeffectuate various AI/ML-based models or decision engines for purposesof some example embodiments of the present patent disclosure that mayinvolve techniques such as support vector machines (SVMs) or supportvector networks (SVNs), pattern recognition, fuzzy logic, neuralnetworks (e.g., ANNs/CNNs), recurrent learning, and the like, as well asunsupervised learning techniques involving untagged data. For example,an SVM/SVN may be provided as a supervised learning model withassociated learning algorithms that analyze data and recognize patternsthat may be used for multivariate classification, cluster analysis,regression analysis, and similar techniques. Given example trainingdatasets (e.g., a training dataset developed from a preprocesseddatabase or imported from some other previously developed databases),each marked as belonging to one or more categories, an SVM/SVN trainingmethodology may be configured to build a model that assigns new examplesinto one category or another, making it a non-probabilistic binarylinear classifier in a binary classification scheme. An SVM model may beconsidered as a representation of the examples as points in space,mapped so that the examples of the separate categories are divided by aclear gap that is as wide as possible (i.e., maximal separation). Newexamples are then mapped into that same space and predicted to belong toa category based on which side of the gap they fall on. In addition toperforming linear classification, SVMs can also be configured to performa non-linear classification using what may be referred to as the “kerneltrick”, implicitly mapping their inputs into high-dimensional featurespaces. In a multiclass SVM, classification may typically be reduced(i.e., “decomposed”) to a plurality of multiple binary classificationschemes. Typical approaches to decompose a single multiclass scheme mayinclude, e.g., (i) one-versus-all classifications; (ii) one-versus-onepair-wise classifications; (iii) directed acyclic graphs; and (iv)error-correcting output codes.

In some arrangements, supervised learning may comprise a type of machineleaning that involves generating a predictive model or engine based ondecision trees built from a training sample to go from observationsabout a plurality of features or attributes and separating the membersof the training sample in an optimal manner according to one or morepredefined indicators. Tree models where a target variable can take adiscrete set of values are referred to as classification trees, withterminal nodes or leaves representing class labels and nodal branchesrepresenting conjunctions of features that lead to the class labels.Decision trees where the target variable can take on continuous valuesare referred to as regression trees. In some other arrangements, anembodiment of the present patent disclosure may advantageously employsupervised learning that involves ensemble techniques where more thanone decision tree (typically, a large set of decision trees) areconstructed. In one variation, a boosted tree technique may be employedby incrementally building an ensemble by training each tree instance toemphasize the training instances previously mis-modeled ormis-classified. In another variation, bootstrap aggregated (i.e.,“bagged”) tree technique may be employed that builds multiple decisiontrees by repeatedly resampling training data with or without replacementof a randomly selected feature or attribute operating as a predictiveclassifier. Accordingly, some example embodiments of the present patentdisclosure may involve a Gradient Boosted Tree (GBT) ensemble of aplurality of regression trees and/or a Random Forest (RF) ensemble of aplurality of classification trees, e.g., in pain score classificationand modeling.

Depending on implementation, various types of data (pre)processingoperations may be effectuated with respect to the myriad pieces of rawdata collected for/from the subject populations, e.g., patients,clinicians, etc., including but not limited to sub-sampling, datacoding/transformation, data conversion, scaling or normalization, datalabeling, and the like, prior to forming one or more appropriatedatasets, which may be provided as an input to a training module, avalidation/testing module, or as an input to a trained decision enginefor facilitating prediction outcomes. In some arrangements, example datasignal (pre)processing methodologies may account for varying timeresolutions of data (e.g., averaging a data signal over a predeterminedtimeframe, e.g., every 10 minutes, for all data variables), missingvalues in data signals, imbalances in data signals, etc., whereintechniques such as spline interpolation method, synthetic minorityover-sampling technique (SMOTE), and the like may be implemented.

In some embodiments, sensor data, video data, and/or audio data areanalyzed during a virtual clinic or remote programming session todetermine rigidity of a patient. For example and referring to FIG. 13 ,a flowchart illustrating exemplary operations that may occur during avirtual clinic or remote programming session according to embodiments ofthe present disclosure is shown. It is noted that the exemplaryoperations shown in and described with respect to FIG. 13 may employvarious types of infrastructure, such as the exemplary infrastructureshown in FIG. 12 , or other infrastructure, such as suitable patientdevice(s), clinician device(s), one or more servers, cloud computingresources, or other types of computing devices/resources suitable forfacilitating the exemplary operations described in more detail withreference to FIG. 13 below.

In the exemplary flow of FIG. 13 , a patient may be instructed toconduct one or more tasks for determining characteristics of thepatient's condition. In some embodiments, the operations of FIG. 13 areconducted concurrently with a virtual clinic and/or remote programmingsession between a patient and a clinician. In some embodiments, theoperations may occur in an automated basis. For example, the patient mayexecute an app on the patient's device to access an “avatar” clinicianwhich instructs the patient to perform the tasks for the patientevaluation. In a patient evaluation session, the avatar clinicianprovides an avatar user interface (UI) to aid a patient on positioningthe camera during the digital exam. The avatar UI may provide an exampleof the exam (e.g., walking, finger tapping, or performance of anysuitable tasks), a target box with color (e.g. about the hands, limbs,torso, head, or other feature to be digitally analyzed), and auditoryfeedback where to move in front of the camera (e.g., for a gait exam).The avatar UI may employ kinematic AI to detect body parts and provideinstructions to position for best camera viewing for the digital exam.

The avatar digital feature of the patient app periodically walks thepatient through standard Parkinson's(movement disorder) exam), whileusing kinematics, and auditory analysis to compare the patient's resultsfrom previous. The avatar patient feature schedules, notifies, andguides the patient through a Parkinson's weekly exam (or other suitabletime period). Kinematics using the camera captures the patients tests asthe avatar clinician in the app instructs the patient how to perform thetasks for the automated exam. The results are captured, and over time atimeline of progression is created and available for the clinician toreview. Further, the patient's clinician is notified if results of anexam fall below limits set.

Exemplary tasks may include flexion tasks, tension tasks, or other typesof tasks that may be used to evaluate the rigidity of the patient. Theone or more tasks may be utilized to capture information that may beutilized to determine characteristics of the patient from which thepatient's condition may be diagnosed. Additionally or alternatively, thecharacteristics may also be utilized to configure a neurostimulationtherapy to address the patient's condition. In some aspects, varioustypes of devices may be employed to gather data during patientperformance of one or more tasks. For example, the devices may includesensors (e.g., accelerometers, gyroscopes, inertial measurement units(IMUs), electro (EMG) sensors, ultrasound sensors, video, and the like).Exemplary sensors that can be employed to obtain data to assist patientevaluation based on patient movement (e.g., during performance of theone or more tasks) are described in the publication “Quantification ofHand Motor Symptoms in Parkinson's Disease: A Proof-of-Principle StudyUsing Inertial and Force Sensors”, Ann Biomed Eng. 2017; 45(10):2423-2436, by Josien C. van den Noort et al., the content of which isincorporated herein by reference.

In some embodiments, a clinician may select one or more tasks forperformance by the patient for the digital exam or evaluation. Forexample, the clinician may select the exams that are appropriate for thepatient at a given time. An avatar feature in the clinician and patientapps will then guide the patient and clinician through the patient tasksfor the digital exam. While the patient tasks are performed, theclinician programmer app and/or one or more server or other computingplatforms performs kinematics and auditory real-time analysis fordisplay to the clinician(s).

In addition or alternative to capturing sensor data as the patientperforms the one or more tasks, one or more cameras may be utilized tocapture media content (e.g., image data, video data, etc.) of thepatient performing the one or more tasks. In some aspects, the sensordata and media content obtained as the patient performs the one or moretasks may be utilized to diagnose the patient's condition during avirtual clinic and/or a remote programming session. In some aspects, aspecific region-based analysis (auto or user-driven) may be conducted toadjust focus within a video field-of view, such as to focus the mediacontent on a particular region of the patient's body (e.g., the hands,legs, feet, upper body, lower body, etc.). The region of interest of thepatient's body may be associated with or determined based on thediagnosed condition(s). To illustrate, a region of interest for apatient diagnosed or suspected of suffering from tremors may be thepatient's hand, and the region of interest for a patient diagnosed orsuspected of suffering from Parkinson's disease may be the patient'sfeet and/or legs (e.g., to perform gait analysis, etc.). It is notedthat the exemplary regions of the patient's body and conditionsdescribed above have been provided for purposes of illustration, ratherthan by way of limitation. Therefore, it should be understood that thediagnostic and analysis techniques disclosed herein may be readilyapplied to other conditions and/or regions of interest on a patient'sbody.

The exemplary flow shown in FIG. 13 may be used to provide real-timefeedback on the patient's condition. This allows changes to bequantified in real-time and associated with neuromodulation or othertherapies, such as by quantifying changes to measure features orcharacteristics over time. This can be achieved through high-definitioncapture and comparison of multiple frames of media content (e.g.,multiple sequential images or frames of video content). Additionally,multiple types of captured data may be utilized to provide multipleindependent data points or to a combination of different data points foranalysis/evaluation (e.g., a combination of captured data associatedwith vascular change data coupled with rigidity data). In aspects wheremedia content is utilized for analysis and evaluation, multiple types ofcameras may be used (e.g., media content may be captured using one ormore imaging cameras, video cameras, and/or thermal-based cameras, suchas infrared cameras) during a session.

As shown in step 1301, the one or more tasks performed by the patientmay include one or more flexion/extension tasks. As the patient performsthe task(s) the system captures patient data using one or more sensors(e.g., IMU sensors, EMG sensors, ultrasound sensors, video camera,and/or the like). In some aspects, the patient data or a portion thereofmay be captured by patient devices, such as the patient devices 104described and illustrated with reference to FIG. 1B.

The one or more tasks performed by the patient may additionally oralternatively include: one or more opposite hand drawing/writing tasks,at step 1302; one or more walking task(s), at step 1303; one or morestanding and/or balancing tasks, at step 1304; and one or morefinger-tapping tasks, as step 1305. In a patient digital examination orevaluation, one or more suitable metrics may be provided to theclinician when a patient performs Bradykinesia finger tapping exam. Theapplication uses kinematics to detect the patient's hand and finger, andthen monitors (distance between the two fingers, and rate of tap). Theapplication provides the clinician a real-time chart showing fingerdistance over time. This chart will let the clinician see the finger tapmovement, and if it slowed up over time.

Any additional tasks may be included as appropriate. For example, thepatient may be instructed to conduct a grip sensing test. For example,using the capacitive touch capacity of a smartphone, the patient's appmay monitor the reported area (based on force) from the smartphone'sdetection of user touch upon the touch screen. Pushing your thumb on thedisplay, the app may monitor the capacitive area reported from the touchpanel matrix. Alternatively, a grip sensor accessory may be employed. Asthe patient grips the device harder more of your hand be detected alongthe side of the smart device. Such sensing may be employed to detectmotor functions and possibly be used to calculate a rigidity score.

As described above with reference to step 1301, various sensors anddevices may be utilized to capture patient data as the patient performsthe one or more tasks. For example, patient data may be captured as thepatient performs the tasks associated with steps 1302-1305 using fingercap accessories, IMU sensors, EMG electrode(s), EEG electrodes,gyroscopes, accelerometers, cameras, temperature sensors, heart ratesensors, blood pressure sensors, other types of sensors, or combinationsthereof (e.g., a combination involving finger cap accessories, IMUsensors, EMG electrode(s), a combination involving finger capaccessories, IMU sensors, EMG and EEG electrode(s), video camera, and soon).

In step 1306, a set of rigidity related metrics are determined. In someaspects, the rigidity related metrics may be determined by one or moreservers, such as a server of the virtual clinic/remote programmingplatform 1214 of FIG. 12 . Each metric may be obtained by processingrespective sets of patient data captured during performance of one ormore of the various patient tasks described above. In some embodiments,the one or more servers may evaluate the patient data to determinecharacteristics associated with the patient's performance of thetask(s), such as a minimum jerk trajectory, a smoothness of motion, aco-contraction profile, a gait profile, an arm-swing profile, abalance/sway, a range of motion, completeness of movement, or othercharacteristics, and the metrics may be determined based on thecharacteristics.

While quantifying metrics around or based on patient movement, stiffnessmeasures or metrics may be calculated. For example, metrics such as theminimum jerk trajectory may be used to compute the smoothness of themovement. In a healthy patient, the movement trajectory has a smoothbell-shaped curve with minimal jerk, which may be calculated based onthe time derivative of acceleration (e.g., based on accelerometer data).When a patient has tremor or rigidity, it is expected that the movementtrajectory deviates from the minimum jerk trajectory (e.g., a jerktrajectory of a healthy patient) and the acceleration may be unsteady.As another non-limiting example, IMU sensors with enough sampling ratecan detect the smoothness of motion based on high frame per second (FPS)video recordings. In addition, in a healthy subject, at the initiationof and during the movement, the co-contraction of the agonist andantagonist muscles decreases. However, in a patient with rigidity, theco-contraction levels are higher at the baseline and the decrease of theco-contraction may not be large enough at the initiation of movement.Muscle tone recordings, such as may be observed using EMGsignals/sensors, of agonist and antagonist muscles may be used alongother sensors for the analysis of the co-contraction profile duringmovement.

Other measures, such as comparison of movement metrics between the rightand left sides (ex: frequency of steps, step interval, or length of armswing) can be used as a proxy for evaluating or quantifying additionalaspects of the patient's condition, such as asymmetry. Metricsassociated with balance can be determined by quantifying sway when thepatient is performing the one or more tasks (e.g., the standing/balancetask(s) at step 1304). Additionally, range-of-motion and completeness ofmovement can be detected when the patient is performing the one or moretasks (e.g., the flexion/extension and/or the walking tasks, at steps1301 and 1303, respectively). Identification of any inability forfull-extension, lack of motion in trunk, abnormal arm-swing, gaitabnormalities (e.g., as detected from lack of heel strike, stoopedposture, movement speed, etc.), or other types of analysis may also beperformed. Other inputs may include proxies for facial expressions asrigidity may also correlate with blank affect.

At step 1307, the respective metrics are used to compute a rigidityscore or other relevant patient scores or measures indicative of thepatient's condition. In some embodiments, the score(s) may be calculatedby one or more servers, such as servers of the virtual clinic/remoteprogramming platform 1214 of FIG. 12 . The calculation of the metricsand/or the rigidity score may be assisted using ML/AI algorithms. Inaspects, the ML/AI algorithms may be trained based on previouslycaptured patient data (e.g., historical patient data captured andprocessed in accordance with the concepts disclosed herein, such as bythe system 1200 of FIG. 12 ).

In some embodiments, a patient-specific method of computing a rigidityscore is performed. For example, Parkinson's Disease (PD) is aheterogeneous disorder, and a score that is patient-specific may be moreinformative for patients and clinicians to understand individual patientprogress (e.g., progression of the disease and/or improvement of thepatient's symptoms as a result of treatment). Therefore, characterizingnoise present in the signal data when the patient is not rigid may beperformed to define a floor value or noise signature that may be usedfor further computations and operations. To illustrate, the noisesignature may be used to characterize the rigidity profile for anindividual patient (e.g., individual patients may have different noisesignatures). When the patient's measures are outside the noise limits,it may serve as an indication of the presence of rigidity (e.g.,rigidity for the specific patient being evaluated).

The above-described techniques may enable the presence of rigidity to betransformed into a binary outcome (e.g., rigidity is present or notpresent on an individual patient basis) that may be further enhancedwith a quantitative score computed from the signals, patient data, andmetrics described above. The ML/AI algorithms may be utilized to createor provide a predictive model that integrates the signals and patientdata collected from sensors and video during performance of the one ormore tasks by the patient. The model may output a value that provides anobjective or sensor-driven score. A subjective or evaluator-driven scorecan also be determined based on clinician assessment andpatient-reported outcomes (PROs). The PROs can also be patient-specificand enhanced by the use of chatbots to ensure that all information isbeing collected without an undue burden on patients. The combinedobjective and subjective scores may represent the overall rigidity scorefor the patient and provide a holistic assessment of the patient'ssymptoms and progress.

In some embodiments, other relevant patient scores may be provided. Forexample, tremor and/or bradykinesia scores may be provided. Alternative,a “pain score” may be provided to the clinician that is related to acomputed level of pain of the patient. The patient score(s) may includethe clinician a real-time chart showing finger distance over time (forthe finger tapping task). This chart will let the clinician see thefinger tap movement, and if it slowed up over time.

At step 1308, the patient score is provided to the clinician. In someembodiments, the rigidity score may be provided to the clinician duringa virtual clinic/remote programming session, such as may be performedusing the system 1200 of FIG. 12 . At step 1309, the patient'sneurostimulation parameters (e.g., deep brain stimulation parameters,spinal cord stimulation parameters, and/or the like) may be modified andused to update the patient's neurostimulation system. As non-limitingexamples, stimulation parameters may include amplitude, frequency, pulsewidth, and the like. It is noted that in some aspects the scorescalculated based on the ML/AI algorithms or model(s) may be utilized toautomatically adjust the patient's neurostimulation parameters, ratherthan providing the score(s) to a clinician and having the clinicianmodify the neurostimulation parameters.

Referring to FIG. 14 , a flow diagram depicting an exemplary flow ofoperations for creating a predictive disorder model using ML/AIprocessing according to some embodiments is shown. Additionally, FIG. 15depicts ML/AI learning data sets 1500 that may be used for creating apredictive disorder model, such as a model created according to theexemplary operations of FIG. 14 . It is noted that the operationsillustrated and described with reference to FIG. 14 may be utilized withdata (e.g., the data captured using the flow of FIG. 13 ) captured usinga virtual clinic/remote programming system (e.g., the virtualclinic/remote programming platform 1214 of FIG. 12 or other systems).

At step 1401, data from neurostimulation patients is captured. In someaspects, the captured data may include data captured from patients in a“stimulation-off” state. Patients may be considered in the“stimulation-off” state when the neurostimulation systems of therespective patients are not providing electrical stimulation to targetneural sites. Referring briefly to FIG. 15 , in some aspects, thecaptured data may include “context” data and “sensor” data. The contextdata may include various types of data, such as geographic data,ethnicity data, disease stage data, existing prescriptions (if any) foreach patient, neurostimulation therapy settings for each patient,timestamps (e.g., time of day when therapy was provided, etc.), andmacro-modality data. It is noted that the exemplary types of contextdata shown in FIG. 15 are provided for purposes of illustration, ratherthan by way of limitation and that the context data may include othertypes of data or combinations thereof. The sensor data, which may becaptured using a variety of sensors or devices, such as peripheralsensors, implanted sensors, or other types of sensors, as described withreference to FIG. 13 . The sensor data may also include image content,video content, audio data content (e.g., captured using camera andmicrophone components of user devices or other types of devices). It isnoted that the sensor data may include data captured using any suitablesensor, device, or component, and that peripheral and implanted sensorsare shown in FIG. 15 for purposes of illustration, rather than by way oflimitation. For example, the sensor data may also include data fromwearable devices, such as biometric or other types of data captured by a“smart” watch, a health monitoring device, or other types of consumerelectronic devices.

Referring back to FIG. 14 , at step 1402, the captured data while thepatients are in the “stimulation-off” may be recorded or stored to oneor more databases, such as one or more databases of a server platform(e.g., the virtual clinic/remote programming platform 1214 of FIG. 12 ).At step 1403, the recorded data is labeled. To illustrate, the datacaptured while the patient is in the “stimulation-off” state may belabeled with information that indicates the data is representative of adisorder state (e.g., patient biometrics, measurements, and the likewere captured in the absence of neurostimulation therapy). The labeleddata may be utilized to train the ML/AI algorithms to recognize thestate of a patient's condition or disorder.

Similarly to steps 14001-1403, data is captured, at step 1404, frompatients in a “stimulation-on” state (e.g., while the patients arereceiving neurostimulation therapy). It is noted that the receiving ofneurostimulation therapy may be continuous (e.g., stimulation pulses maybe delivered to target tissue of the patient continuously during thedata capturing at step 1404) or periodic (e.g., stimulation pulses maybe delivered to target tissue of the patient for a period of time andnot delivered for a period of during the data capturing at step 1404).At step 1405, the captured data, which may include image data, audiodata, video data, sensor data, other types of data, or combinationsthereof, is recorded to one or more databases. At step 1406, the datacaptured while the patient is in the “stimulation-on” state may belabeled with information that indicates the data is representative of astate of the patient's condition during providing of theneurostimulation therapy (e.g., patient biometrics, measurements, andthe like were captured in the presence of the neurostimulation therapy).The labeled data may be utilized to train the ML/AI algorithms torecognize the state of a patient's condition or disorder whenneurostimulation therapy is provided or the impact of theneurostimulation therapy on the patient's condition or disorder. At step1407, the state/context data (e.g., the state/context data of FIG. 15 )may be processed or conditioned (e.g., using generalized Bayeconditionalization, Jeffrey conditionalization, and/or any othersuitable methods). The conditioning may improve model accuracy.

At step 1408, ML/AI processing is applied to at least a portion of thecaptured data (e.g., a portion of the data captured while the patientsare in the “stimulation-off” and “stimulation-on” states) to develop apredictive disorder model. In some aspects, the predictive disordermodel may be configured to classify disorders of patients (e.g., healthstate, disorder state, disorder level, etc.). The model(s) may utilizevarious classification techniques, such as random forest classification,Naive Bayes classification, k-means clustering, genetic algorithms,neural networks, reinforcement learning strategies (e.g., Q-learning,Temporal Difference learning, Markov decision processes, etc.), and/orany suitable ML/AI methods.

The predictive disorder model may include various disorder states. Forexample, for a movement disorder such as Parkinson's Disease, thepredictive disorder model may include model components such as tremor,rigidity, facial drop, balance, hallucinations, and/or any otherrelevant disorder symptoms. As illustrated in FIG. 15 , the modelcomponents may also be employed to create an overall disorderclassification or classification model, such as “PD level” that reflectsan ensemble of the selected model components. It is noted that whileFIG. 15 illustrates aspects of a model for a movement disorder, suchdescription has been provided for purposes of illustration, rather thanby way of limitation and predictive models of the present disclosure mayutilize additional model data for analysis of other disorders, such aschronic pain disorders.

At step 1408, the predictive model is used to evaluate patients andascertain the condition(s) of their neurological disorders. As describedabove, the predictive models may be trained with datasets associatedwith patients in the “stimulation-off” and “stimulation-on” states. Assuch, the predictive models may be utilized to evaluate the condition ofa patient's disorder or condition based on data captured while theneurostimulation stimulation therapy (or other type of therapy) is or isnot being provided. The outputs of the predictive model may be utilized(e.g., by a clinician or automated tools) to assist with programming ofneurostimulation parameters (e.g., adjust the parameters, conditions fortriggering delivery of neurostimulation, etc.), augmenting virtualclinic/remote programming sessions, and/or any other suitable patientevaluation processes.

As discussed herein, video data of patients may be processed to supportproviding neurostimulation therapies according to some embodiments. Theprocessing of the video data may occur substantially in real-time (e.g.,during a virtual clinic/remote programming session). Alternatively, theprocessing of video data may occur on previously recorded video. Theprocessing of the video data may be used to assist evaluation of one ormore symptoms of the patient. Additionally or alternatively, theprocessing of the video data may be utilized to build a ML/AI model foruse in neurostimulation therapies.

Referring to FIG. 16 , a flowchart depicting exemplary operations forprocessing video data to support provision of neurostimulation therapiesto patient according to some embodiments is shown. At step 1601, videodata of the patient is captured. The video data may be captured, forexample, during a virtual clinic/remote programming session according tosome embodiments. The video data may be captured from a user device,such as patient controller (e.g., patient controller 150 of FIG. 1A,patient controller devices 1210 of FIG. 12 , or other types of devices).In step 1602, the video data is processed to extract features associatedwith the patient. To illustrate, the extracted features may beassociated with the patient's facial expression, posture, limb activity,or other anatomical considerations. In some aspects, the features may beextracted using landmark recognition software. For example, if the videoanalysis is conducting an analysis of facial expression, suitablesoftware libraries include the FACEMARK facial landmark API availablefrom OpenCV, the CLM-framework (also known as the Cambridge FaceTracker), FACE++ Web API, CLOUD VISION API (Google) as examples. Whenthe video analysis is conducted based on posture, limb activity, orother anatomical considerations that involve a large amount of thepatient's body, other landmark software packages may be employed such asMEDIAPIPE Pose (Google) and OPENPOSE library (OpenCV). MediaPipe alsoprovides a number of software API packages for video analysis includingFACE MESH and POSE packages for detecting facial landmarks and bodylandmarks respectively.

In step 1603, key landmark points are selected from the points orfeatures generated during step 1602. The key landmark points may beselected to identify relevant characteristics related to neurologicaldisorder symptoms. For example, facial expression and body posture canbe related to pain and/or motor symptoms of chronic neurologicaldisorders. For example, FIGS. 17A and 17B depict sets 1700 and 1750,respectively, of key landmark points that may be utilized for facialexpression analysis and pose analysis according to some embodiments.

In step 1604, regional area metrics are calculated from the key landmarkpoints. The regional area metrics are indicative of the area bounded bykey points. For example, in FIG. 17A, areas A_(LEFT) and A_(RIGHT) areshown, and in FIG. 17B areas A1-A4 are shown. As discussed herein, thecalculated areas may be compared to generate a representation of balancein expression and/or pose which is related to symptoms of neurologicaldisorders. To illustrate, when a patient is comfortable and happy, theshape of the face is balanced and exhibits a relatively high degree ofsymmetry. However, when a patient is subjectively experiencing pain,their facial expression will often contort and lose the balance orsymmetry in the shape of the face. Similarly, a loss of balance orsymmetry can be detected in the pose of a patient when the patient isexperiencing pain or is subject to certain symptoms of motor disorders(e.g., Parkinson's Disease). In some embodiments, the patient may berequested to perform one or more physical task (e.g., as described withreference to FIG. 13 ) while the data is captured and/or the analysisoccurs. For example, the patient may be requested to walk a shortdistance within the view of the patient's camera. The processing andanalysis of the patient's pose or posture during this task may occur toclassify the patient's disorder state.

In step 1605, one or more metrics may be calculated based on the set oflandmark points or features and/or regional areas. For example, the oneor more metrics may include ratios, such as a balance ratio, a crossratio, and/or other relevant ratios derived from the landmark pointsand/or regional areas. In some aspects, the ratios and/or metrics may becalculated for each video frame or for a relevant fraction or set ofvideo frames. For example, one frame every 0.1 or 0.2 seconds may beselected for analysis even though the frame rate of the video signal maybe higher. The selection of the frame rate analysis may be varieddepending on the quality of the video signal and any other relevantfactor.

To illustrate, analysis of a patient's facial expression using the setof key landmark points 1700 of FIG. 17A, the two areas (A_(left) andA_(right)) may be defined based on the respective key landmark points asshown in FIG. 17A. The balance ratio may be defined as:

balance ratio=A _(left) /A _(right).

For analysis of facial expression using the set of key landmark points1700, a distance (D_(vertical)) between point 10 and point 9 and adistance (D_(horizontal)) between point 4 and point 5 may be determined.Subsequently, D_(vertical) and D_(horizontal) may be used to calculate aface ratio. The face ratio may be expressed as:

face ratio=D _(vertical) /D _(horizontal).

For analysis of facial expression using set of key landmark points 1700,a distance (D_(cross1)) between point 11 and point 14, and a distance(D_(cross2)) between point 12 and point 13 may be calculated and used todetermine a cross ratio. The cross ratio may be defined as:

Cross ratio=D _(cross1) /D _(cross2).

With reference to FIG. 17B, analysis of patient pose or posture usingthe set of key landmark points 1750, let D_(left) be defined as thedistance between point 1 and point 7, and D_(right) be defined as thedistance between point 2 and point 8. Using D_(left) and D_(right), abalance ratio may be determined. The balance ratio may be defined as:

Balance ratio=D _(left) /D _(right).

For analysis of patient pose or posture using the set of key landmarkpoints 1750, let D_(cross1) is defined as the distance between point 1and point 8, and D_(cross2) be defined as the distance between point 2and point 7. The cross ratio may then be defined as:

cross ratio=D _(cross1) /D _(cross2).

For analysis of patient pose or posture using set of key landmark points1750, let D_(shoulder-knee1) be defined as the distance between point 1and point 5 and let D_(shoulder-knee2) be defined as the distancebetween point 2 and point 6. A shoulder-knee ratio may then becalculated based on D_(shoulder-knee1) and D_(shoulder-knee2). Forexample, the shoulder-knee ratio may be defined as:

shoulder-knee ratio=D _(shoulder-knee1) /D _(shoulder-knee2).

For analysis of patient pose or posture using set of key landmark points1750, let D_(elbow-foot1) be defined as the distance between point 9 andpoint 8, and let D_(elbow-foot2) be defined as the distance betweenpoint 10 and point 7. An elbow-foot-cross ratio may be calculated basedon D_(elbow-foot1) and D_(elbow-foot2). For example, theelbow-foot-cross ratio may be defined as:

elbow-foot-cross ratio=D _(elbow-foot1) /D _(elbow-foot2).

Referring back to FIG. 16 , at step 1606, a time domain sliding windowis applied to the one or more metrics (e.g., the ratio data and/or otherrelevant metric data). In some embodiments, the sliding window may havea duration of one second. That is, the ratio and metric data from agiven one second time window is selected for aggregate analysis.However, it is noted that sliding windows having a duration longer thanone second or shorter than one second may be utilized.

In step 1608, a ML/AI model, such as the predictive model describedabove with reference to FIGS. 14 and 15 or another model, may be appliedto media content of a particular sliding window. The model may beconfigured to process or apply a classification the media content (e.g.,to frames of video content, etc.). In an aspect, only relevant portionsof the media content (e.g., relevant video frames) within the slidingtime window may be classified. For example, the relevant portions mayinclude video frames determined to exhibit (e.g., by the model) athreshold probability of being correctly classified as either depictinga healthy/normal state or a symptom state (e.g., pain).

The model may be configured to calculate a probability of relevance foreach portion of the media content (e.g., a probability that aclassification for a particular frame of video depicts a healthy/normalstate or a symptom state), and relevant portions of the media contentmay be identified based on a threshold probability. In an aspect, thethreshold probability used to identify relevant portions of the mediacontent may be a probability of at least 0.8 (e.g., media content havinga probability of relevance 0.8, where 0.8=threshold probability). It isnoted that a threshold probability of 0.8 has been described forpurposes of illustration, rather than by way of limitation and that thethreshold probability may be configured to a higher or lower value ifdesired. Moreover, in some aspects, different threshold probabilitiesmay be used for different conditions/symptoms. Regardless of theparticular value to which the threshold probability is set or the numberof probability threshold used, it should be understood that thethreshold probabilities may be changed over time. For example, initiallythe threshold probability may be configured to a first value (e.g.,0.65), but the model may become more accurate over time (e.g., asadditional training data is obtained and additional training of themodel is performed). As the model's classification capabilities improvethe threshold probability may be adjusted, thereby minimizing thelikelihood that portions of the media content identified as relevant endup being unsuitable for use in evaluating the patient. It is noted thatany portions of the media content having a threshold below the thresholdprobability may be ignored (e.g., because those portions of the mediacontent may be associated with inaccurate or incorrect classifications,or may otherwise be unsuitable for further use in evaluating the stateof the patient or the patient's condition).

At step 1609, the relevant portions of the media content identified instep 1608 may be used to calculate a disorder score. In an aspect, thedisorder score may be calculated based on metrics derived from therelevant portions of the media content, such as the exemplary metricsdescribed above with reference to step 1604. In some aspects, aggregatemetrics (e.g., average, median, etc.) may be calculated based on theratios/metrics of the relevant portions of the media content. Thecalculated ratio/metrics for the relevant portions of the media contentwithin the sliding window may then be subjected to ML/AI processing orclassification to calculate the disorder score, which may be attributedto or associated with the sliding window (and the relevant portions ofthe media content thereof).

In step 1609, the disorder score(s) is provided to the clinician. Insome aspects, the disorder score(s) may be provided to the clinician viaa graphical user interface, such as a graphical user interface of anapplication resident on a clinician programmer device. In some aspects,the graphical user interface may include one or more graphical userinterface components that change color in a manner relevant to thepatient state classification. For example, the patient video componentmay include a border component. The border component may include a“green” color for a patient normal state, a “red” color for a patientsymptom present state (e.g., pain, rigidity, tremor, and/or the like ispresent), and a “neutral” color (e.g., gray) for intermediate oruncertain classifications. It is noted that the exemplary colors andassociated meanings described above have been provided for purposes ofillustration, rather than by way of limitation and that other types ofcolor schemes and indications may be utilized in accordance with theconcepts disclosed herein.

In step 1610, the clinician or a computational therapy algorithmprovides settings for the patient's neurostimulation therapy based onthe processed video of the patient. In some aspects, the above-describedprocess may be performed continuously or repeatedly during the virtualclinic/remote programming session. That is, as the clinician changesstimulation parameters, the indication of the patient state is updatedas the video of the patient is streamed to the clinician for reviewduring the virtual clinic/remote programming session.

As discussed herein, ML/AI models of neurological disorders may beconstructed using a variety of data sets. The ML/AI models may beemployed to automatically classify patient states to assist virtualclinic/remote programming sessions. A subjective or evaluator-drivenscore can also be determined based on clinician assessment andpatient-reported outcomes (PROs). The PROs can also be patient-specificand enhanced by the use of chatbots to ensure that all information isbeing collected without an undue burden on patients. ML/AI models may beconstructed using, in part, such data. Such data may be obtained priorto a virtual clinic/remote programming session, at its initiation, orduring the session. For example, patient emotional/well-being data maybe obtained at the beginning of a session to increase the accuracy ofthe ML/AI operations during the session. Other models may be constructedfor use or selection by the clinician based on one or more of the datatypes described herein (e.g., video, audio, sensor, context, patientreported data, PROs data, and/or any other PAD as discussed herein). Theclinician may select from available ML/AI models and/or the virtualclinic/remote programming infrastructure or CP app may automaticallyselect the appropriate ML/AI model(s) based on available data. In someembodiments, the virtual clinic/remote programming infrastructure or CPapp may select appropriate models for use during a remote-programmingsession based on latency or other context. For example, different modelsmay be selected depending upon a task being performed by the patient.Audio only models may be applied at selected portions of theremote-programming session (e.g., during clinician interview of thepatient) and other models at different times (e.g., during patientperformance of physical tasks). Also, certain models mayactivated/deactivated based on available processing resources andlatency constraints associated with AN session.

The respective ML/AI models may be employed to automatically classify orquantify patient states to assist virtual clinic/remote programmingsessions. Referring to FIG. 18A, a screenshot depicting exemplary userinterface for a clinician programmer device for conducting virtualclinic/remote programming sessions is shown as a user interface 1800.The user interface 1800 may be adapted to provide analytics of thepatient concurrently with presentation of video content of the patientas the patient performs one or more tasks (e.g., the one or more tasksof FIG. 13 or other tasks) to the clinician. As described above, thepresentation of the video content may include presentation ofpre-recorded video of the patient. Additionally or alternatively, thepresentation of the video content may occur in real-time, such as viastreaming the video content to the clinician programmer device during alive virtual clinic/remote programming session.

In some embodiments, the analytics include analytics associated withkinematics (e.g., data related to movement of the patient). In someembodiments, the analytics include auditory data from analysis of thepatient's voice. It is noted that the exemplary analytics describedabove (e.g., kinematics and auditory analytics) have been provided forpurposes of illustration, rather than by way of limitation and thatadditional types of analytics may be utilized by embodiments of thepresent disclosure. The user interface 1800 may include interactive GUIelements that enable the clinician to control selection of availableones of the analytics during presentation the media content. Forexample, the user interface 1800 may include pop-up control component1801. The pop-up control component 1801 may allow the clinician toactivate selected analytics, as well as configure parameters associatedwith the analytics. To illustrate, in FIG. 18A the pop-up controlcomponent 1801 includes interactive elements (e.g., check boxes) thatenable the clinician to select specific kinematics, such as kinematicsrelated to the patient's face, hands, arms, and legs. The pop-up controlcomponent 1801 also includes interactive elements to activatepresentation of analytics associated with the patient's voice.

As described above, the media content (e.g., video data) may beprocessed using feature recognition software to identify variousfeatures or points associated with the patient. The features may bedisplayed over the patient video during presentation of the mediacontent in accordance with the analytics activated by the clinician. Insome aspects, lines connecting different features (e.g., key landmarkpoints) may also be displayed as appropriate (e.g., along the torso,arms, legs, fingers, etc.) as shown in interface 1800. Additionally, amesh display generated based on facial features may be displayed overthe patient video as shown in interface 1800. In some aspects, the meshdisplay may be generated using recognition software libraries, such as alibrary or libraries of the above-mentioned landmark recognitionsoftware.

In some embodiments, the presentation of video and/or audio may beanonymized. As shown in FIG. 18B, video of the patient may be replacedentirely by display of the landmark points, mesh, and frame componentsgenerated based on the features identified in the media content. It isnoted that the exemplary landmark points, mesh, and frame components maybe animated based on the kinematics analytics and/or the auditoryanalytics, thereby enabling the clinician to visualize movement of thepatient as the patient performs the one or more tasks, such as the tasksdescribed above with reference to FIG. 13 or other tasks. It is notedthat in some aspects the user interface 1800 may also provideinteractive elements that enable the clinician to toggle between theoverlay view described with reference to FIG. 18A and the anonymous viewdescribed with reference to FIG. 18B. The ability to toggle between thetwo different views may enable the clinician to more easily see certainkinematics or focus on specific aspects of the displayed kinematics(e.g., to display kinematics of the patient's hands and arms duringevaluation of tremors, or feet and legs during evaluation ofParkinson's, such as gait analysis).

Referring to FIG. 18C, a screenshot depicting aspects of a userinterface which may be displayed on a patient device is shown as a userinterface 1820. In some aspects, the user interface 1820 may bedisplayed on a patient controller device (e.g., the patient controller150 of FIG. 1A, patient controller 800 of FIG. 8 , or patient controller1210 of FIG. 12 ) or another patient device (e.g., one or more of thepatient devices 104 of FIG. 1B). The user interface 1820 may includefunctionality for generating media content (e.g., video content, imagecontent, and/or audio content) that may be used in connection with thevarious analytics described above with reference to FIGS. 18A and 18B.

As shown in FIG. 18C, the user interface 1820 may display the landmarkgraphical features corresponding to the kinematic components selected bythe clinician, enabling the patient to view the same kinematics as theclinician. Additionally, user interface 1820 may be used to providefeedback or commands to the patient, such as instructions for thepatient to move their face, hands, limbs, torso, etc. into an optimalposition for generation of the selected analytics. For example, userinterface 1820 includes GUI component 1821, shown as a box, around thepatient's right hand. In some aspects, the GUI component 1821 mayprovide a visual indication regarding whether the patient or a portionof the patient's body, is in a desired or optimal visual location. Forexample, the GUI component 1821 may be colored “green” to indicate thepatient is in the optimal location and colored “red” to indicate theneed to reposition. Further, the GUI component 1821 may include one ormore indicators, shown as an arrow in FIG. 18C, to indicate a directionof movement to place the patient into the optimal location.

Referring to FIG. 18D, a screenshot depicting aspects of a userinterface presented to a patient are shown as GUI component 1830. TheGUI component 1830 may provide functionality for controlling how inputfrom the patient during a virtual clinic/remote programming session isreceived. For example, GUI 1830 may include interactive elements (e.g.,check boxes, radio buttons, etc.) that may be selected or activated bythe patient to grant permission for audio communication, visualcommunication, or audio/visual communication during the session.Additionally, the GUI component 1830 may provide interactive elementsenabling the patient to control whether auditory and/or video contentmay be recorded during the session.

The GUI component 1830 may also include interactive elements that enablethe patient to control whether the kinematic analytics are displayed tothe clinician in the anonymized or non-anonymized mode, includingsub-segments. For example, the patient may select to anonymize all videodata by checking the interactive element labeled “Video” underneath thefirst “Anonymize” header. In such case, all available video options willbe automatically checked. Alternatively, the patient may select specificvideo sub-components to be anonymized (e.g., “Face”, “Hands”, “Arms”,“Legs,” “Torso” and “Background). Similarly, the patient may also chooseto anonymize the voice communication (if desired). The patient maychoose to allow the clinician to view a full video presentation (such asthe patient view in FIG. 18C, a completely anonymized view such as thepatient view in FIG. 18B, or any variation thereof). It is noted thatthe GUI component 1830 is provided by way of illustration, rather thanby way of limitation and that the GUI 1820 may include other suitableinterface configurations and components that provide functionality forallowing the patient to view and control aspects of audio and visualcommunications during a virtual clinic/remote programming session.

Referring to FIG. 19 , a flowchart depicting exemplary operations forconducting a virtual clinic/remote programming session according to someembodiments is shown. In step 1901, a virtual clinic session is started(e.g., using the operations discussed herein). In step 1902, thepatient's consent for AV communications is obtained. Obtaining thepatient's consent may include obtaining user input for respectiveanonymizing options, as described above with reference to FIGS. 18C and18D. In some aspects, the patient (or the clinician) may choose toremove the background from the video communications or any bodily regionor area from the video (including, optionally, one or more of theoptions from FIG. 18D). In step 1903, the initial AV communication forthe virtual clinic/remote programming session begins according to thepatient AV selections. In step 1904, options for kinematic and/orauditory analysis are received from the clinician to assist patientevaluation, as described above with reference to FIGS. 18A and 18B. Instep 1905, the video/audio stream is augmented with ML/AI assistedkinematic and/or auditory analytics (including, optionally, one or moreof the analytics discussed herein). In step 1906, the clinician or acomputational therapy algorithm provides settings for the patient'sneurostimulation therapy based on the processed video of the patient.

As discussed herein, some embodiments the kinematic analysis may beconducted while a patient performs a functional task (e.g., walking topermit evaluation of the patient by the clinician). In some embodiments,a patient physical therapy application is provided to the patient toassist management of the patient's neurological disorder. The physicaltherapy application may operate on a patient controller (e.g., thepatient controller 150 of FIG. 1A, patient controller 800 of FIG. 8 , orpatient controller 1210 of FIG. 12 ) or another patient device (e.g.,one or more of the patient devices 104 of FIG. 1B). In some embodiments,virtual reality (VR) devices may be used to assist the physical therapyoperations. The physical therapy application may provide guidedinstructions for the patient to conduct physical activities. Thephysical activities may include activities selected to improve thepatient's health, improve motor function, accommodate the patient tochanges in physical sensations during movement as a result of aneurostimulation therapy, and/or assist any suitable clinicalimprovement in the patient. Additionally, the physical therapyapplication may capture patient data while the patient performsrespective activities according to the instructions of the app. Forexample, physiological data, movement data, and other data may becaptured using one or more patient devices (e.g., one or more of thepatient devices 104 of FIG. 1B or another device). Additionally, videoand/or audio data from the patient may be captured and analyzed asdiscussed herein. The captured data may be reviewed by the clinician toevaluate the patient's response and progress to neurostimulationtherapy.

Referring to FIG. 20 , a flowchart depicting exemplary operations forconducting a physical therapy session in accordance with someembodiments is shown. In step 2001, the patient starts the physicaltherapy application on a patient controller (e.g., the patientcontroller 150 of FIG. 1A, patient controller 800 of FIG. 8 , or patientcontroller 1210 of FIG. 12 ) or another patient device (e.g., one ormore of the patient devices 104 of FIG. 1B). In some aspects, thephysical therapy application may be the physical therapy applicationdescribed above. In step 2002, patient activity is recorded using camerafunctionality and/or sensors of available patient devices. As explainedabove, media content may be recorded during the patient's performance ofthe task(s) or activities using camera functionality and/or a microphonefunctionality of the patient's patient controller device, computer, orother suitable device. In some aspects, one or more external sensors mayadditionally or alternatively be employed to gather relevant patientdata. For example, insole pressure sensors for quantifying balance,pressure distribution, or other information may be utilized. Also,wearable accelerometers and/or gyroscopes (e.g., embedded withinbracelets, watches, gloves, shoes, and the like) may be used to captureinformation that may be used to quantify arm and leg movement, sleep,trajectory, and/or timing. In some aspects, the patients can opt to wear“markers” on key joints of their body and use an optic movement-trackingsystem (such as Kinect). In some embodiments, breathing sensors, such asstrap-on chest pressure sensors may be employed to detect respirationactivity. Also, any number of sensors for sensing cardiac relatedactivity may be employed (e.g., heart rate/blood oxygen sensors, such aswatches, or wearable EKG sensors to detect heart rate and oxygen level).

In step 2003, the patient is provided guided instructions for one ormore physical tasks or activities to be completed by the patient. Insome aspects, the guided instructions may include video and/or audiopresentations. For example, a video or images and text may be displayedto the user to illustrate the types of tasks or activities the patientis to perform.

Patients with movement disorders such as Parkinson's disease oftenreport difficulties with everyday tasks such as buttoning (e.g., ashirt), brushing, and/or writing. In some embodiments, the physicaltherapy application may be tailored to the specific condition ordisorder of the patient in order to train the patient on activitiesimpacted by their specific condition or disorder, which may providesubstantial improvements in the patient's quality of life.

In some embodiments, the physical therapy application may provideinstructions for suitable physical exercises, such as tai chi or mild tomoderate effort treadmill training, aerobic training, and danceactivities. Tai chi may be advantageous for patients with neurologicaldisorders. There have also been clinical studies that investigated thebenefit of exercising, particularly the ones that involve balancetraining, such as tai chi, for movement disorders (e.g., Parkinson'sdisease). For example, in a study (Tai Chi versus routine exercise inpatients with early- or mild-stage Parkinson's disease: a retrospectivecohort analysis, Braz J Med Biol Res. 2020; 53(2): e9171) that involved500 people with mild-to-moderate Parkinson's disease, one group receivedtai chi training 80 minutes per day, three days per week, for twomonths. The other group received regular exercising (including treadmilltraining, aerobic training, and dance) for 90 minutes per day, threedays per week, for two months. Participants in the tai chi groupreported a significantly reduced number of falls (average of 3.45 vs.7.45 over the past six months), and many of them discontinued or reducedthe use of other therapies, such as levodopa.

Similarly, in another study (Tai Chi and Postural Stability in Patientswith Parkinson's Disease, N Engl J Med 2012; 366:511-519) that recruited195 men and women with mild to moderate Parkinson's disease, subjectswere randomly assigned to twice-weekly sessions of either tai chi,strength-building exercises, or stretching. After six months, those whodid tai chi were about two times better than those in theresistance-training group and four times better than those in thestretching group in terms of balance. The tai chi group also hadsignificantly fewer falls, and slower rates of decline in overall motorcontrol. These studies demonstrate that exercise that involves balancetraining as part of a physical therapy routine can provide additionalbenefits for people with balance/gait-related disorders, such asParkinson's disease, in addition to the benefits exercising itselfalready brings.

In some embodiments, the patient may be provided a VR or AR viewingdevice to augment the user experience for the presentation of the guidedinstructions (and feedback described herein). Although optional, beingimmersed in a VR/AR-based environment can often encourage the trainee toconsistently exercise by providing additional visual and audio stimuli.In the case of tai chi training, a pre-recorded VR/AR teacher can bepresented in front of the trainee for learning. The VR/AR teacher canalso be a persona (or hologram) based on adaptable chatbots topersonalize the therapy experience. The teacher's body can besuperimposed onto the trainee's body so that the trainee can mimic theteacher's exact movements. Here, the trainee would match the idealmovement trajectory/posture outlined by the digital teacher. In thiscase, the gaze of the patient or a slight change of EMG may cue themovement initiation.

In another aspect of exercising with a virtual reality or augmentedreality experience according to the present disclosure, the VR or ARdetects the intention of the subject, filters out the tremor, anddisplays undisturbed arm or leg movement. The rationale is that a tremorin Parkinson's disease patients may be caused by overcompensation of theposture in the body control while the postural information is erroneousbecause of the malfunction of thalamic relay neurons. By displaying thecorrect posture without the tremor using VR/AR, the patient may stoptrying to compensate (or overcompensate) for the postural error, whichhypothetically may reduce the occurrence of the tremor or the severityof the tremor. Because the tremor frequency is around 5 Hz, notchfiltering may remove the tremor and show smooth motion.

In case of a freezing event, the display of the intended movementinitiation may rescue a patient from freezing.

In some embodiments, a gesture training paradigm may also be implementedfor patients who prefer gesture training over (or in addition to)balance and gait training. This training will leverage music or art“therapy.” In other embodiments, the gesture training paradigm can alsobe disguised as games instead of music training. In embodiments using amusic-based therapy paradigm, the patient will learn to control thenotes and the pitch of the music via various common gestures they woulduse in real life, such as brushing, writing, buttoning, fingermovements, and the like. Each gesture can code a note, and the relativepositioning of the two hands/arms can control pitch. This paradigm couldlook very similar to playing the theremin, and an AI may be implementedto rate the performance and offer haptic feedback (as discussed herein).Gloves or arm sleeves with actuators embedded at the interface may beemployed for haptic feedback in some embodiments.

A less specific version of this gesture training can also be implementedas “3D painting”, where the patient is asked to paint with a VR devicein a virtual 3D space. This would involve precise positioning of finemovements as well as localization in a 3D space. This can be done eithervia copying an existing 3D painting, or having the patient create theiroriginal painting. In either case, an AI can be used to judge themovement precision of the patients, instead of the paintings themselves.

In step 2004, feedback may be provided during patient performance of thetask(s) or activities. Having this measurement in turn offersopportunities for the AI in the physical therapy application to offerfeedback to the patient if the patient repetitively makes the wrong moveor mistimes the move. The feedback can be achieved via actuatorsembedded into the fabrics of the patient's clothing, or special clothingarticles (such as gloves, socks, shoes) that have embedded actuators.For example, if the patient is supposed to shift his weight to the leftfoot but failed to do so, a vibration can take place on the left foot toremind the patient. The cadence and frequency of the feedback can bedriven by the AI or set by the patient. Presumably, as the patient getsbetter via practicing, the haptic feedback can change to a differentvibration pattern to signal more complex feedback, such as to acceleratemovements, deaccelerate movements, or even signaling a “good job”.

In some aspects, exercise programs (moves) may have different difficultylevels. For patients who have more severe disease conditions or fasterprogressive severity, the program may start with the easiest level(s)and work their way up to higher difficulties. The AI can alsocorrespondingly offer more feedback and support as the exercisedifficulties increase.

In the case of a fall, or any situation of crisis, the sensors beconfigured to detect the adverse event and offer an “emergencyintervention” by either switching to the most efficacious setting, orcalling a care coordinator. Similarly, if the sensors detect that thepatient is engaging in more dangerous movements, strong vibrations canalso be sent via haptic feedback to remind/prevent the patient fromperforming movements that are out of the patient's comfort zone.

In step 2005, the stimulation parameters of the patient'sneurostimulation therapy may be titrated during the performance of thetasks or activities. For example, if a patient has an implanted devicesuch as DBS, the device can interact with the exercise platform in aclosed-loop manner. The efficacy of neurostimulation for a neurologicaldisorder can be state-dependent, as exercising could potentially changethe efficacy of certain programmed settings. In such a situation, theimplanted device can be controlled (e.g., by the patient controllerdevice) to make small adjustments to programmed parameters to “explore”the therapeutic state space when patient performs various exercises, andthe data can be used as training data for a deep learning algorithm topredict which parameter set is best suited for each exercise for thisparticular patient, thereby enabling an “exercise mode” to beindividually developed for each patient. This can also includeexplorations of known and/or novel stimulation waveforms and paradigmsthat could be better suited for the patient given a specific exercise.

Additionally, based on how well the patient is progressing in theirfamiliarity and control of their exercise, an AI can be trained toslowly decrease the current of the implanted device setting, andtherefore offering less therapy. This will offer the patient more“challenge” in terms of not being able to control their movement due tomovement disorder symptoms. This “rehab mode” can motivate the patientto be intentional with their exercise training. The percent deviationfrom the optimal programming parameter can be prescribed by a physicianand be controlled within a configured safety limit based on theparticular patient, such as to maintain the parameters within atherapeutic range. In this manner, a digital equivalent of physicaltherapy may be provided and enable patients to perform therapy tasks atincreasing levels of difficulty (either due to the particular tasksselected, the speed at which the tasks are performed, or due toparticular configurations of stimulation parameters).

Similarly, if the patient performs worse in the exercise over time, theAI can adjust to a more efficacious therapeutic amplitude or setting, tooffer more assistance to the patients.

Further, exercise alters the plasticity of the brain, and therefore,with long term recordings of sensor data and neural data at theimplanted devices, one can infer any long term changes in the patient'sdisease improvement/progression. A correlation can be helpful for theclinicians to understand the patient's disease states and can alsoinstruct a change in the programming setting of the implanted device.

In step 2006, an analysis of patient video and/or sensor data isconducted to characterize the patient conducting. The analysis mayinclude ML/AI processing as discussed herein. Kinematic data may becalculated or determined for the patient based on the video and sensordata using one or more of the techniques described herein.

In step 2007, the performance data and/or video data is provided to aclinician for review. For example and referring to FIG. 21 , ascreenshot of an exemplary graphical user interface for presentingperformance data associated with a patient is shown as a user interface2021. The user interface 2021 may be configured to present video of thepatient conducting a task or activity (e.g., tai chi) with kinematicanalysis as discussed herein. The video may be provided to the clinicianafter being previously recorded by the patient. Additionally oralternatively, the video may be provided to the clinician concurrentlywith the patient performance of the activity (e.g., streaming the videocontent during a virtual clinic/remote programming session). If deemedappropriate (e.g., by the clinician viewing the video of the patient oran AI/ML process), new or adjusted stimulation therapy settings may beprovided for control the patient's neurostimulation system, in step2008.

Although the present invention and its advantages have been described indetail, it should be understood that various changes, substitutions andalterations can be made herein without departing from the spirit andscope of the invention as defined by the appended claims. Moreover, thescope of the present application is not intended to be limited to theparticular embodiments of the process, machine, manufacture, compositionof matter, means, methods and steps described in the specification. Asone of ordinary skill in the art will readily appreciate from thedisclosure of the present invention, processes, machines, manufacture,compositions of matter, means, methods, or steps, presently existing orlater to be developed that perform substantially the same function orachieve substantially the same result as the corresponding embodimentsdescribed herein may be utilized according to the present invention.Accordingly, the appended claims are intended to include within theirscope such processes, machines, manufacture, compositions of matter,means, methods, or steps.

Moreover, the scope of the present application is not intended to belimited to the particular embodiments of the process, machine,manufacture, composition of matter, means, methods and steps describedin the specification.

1. A method of remotely programming an implantable medical device thatprovides therapy to a patient, comprising: establishing a firstcommunication between a patient controller (PC) device and theimplantable medical device, wherein the implantable medical deviceprovides therapy to the patient according to one or more programmableparameters, the PC device communicates signals to the implantablemedical device to set or modify the one or more programmable parameters,and the PC device comprises a video camera; establishing a videoconnection between the PC device and a clinician programmer (CP) deviceof a clinician for a remote programming session in a secondcommunication that includes an audio/video (A/V) session; communicatinga value for a respective programmable parameter of the medical devicefrom the CP device to the PC device during the remote programmingsession; and modifying, by the PC device, the respective programmingparameter of the medical device according to the communicated value fromthe CP device during the remote programming session; wherein the methodfurther comprises: receiving sensor data from a wearable device of thepatient that includes sensed physiological data; automaticallyanalyzing, by one or more processors, the sensor data and historicalpatient data to dynamically generate one or more metrics related to aneurological condition of the patient during the remote programmingsession; and augmenting video of the patient in the NV session with oneor more graphical user interface (GUI) elements for display on the CPdevice to indicate a level or classification of the one or more metricsrelated to the neurological condition of the patient.
 2. The method ofclaim 1 wherein the sensor data comprises data obtained from asmartwatch device of the patient.
 3. The method of claim 1 wherein thesensor data comprises data obtained from a sensor implanted within abody of the patient.
 4. The method of claim 1 wherein the historicalpatient data includes baseline patient data.
 5. The method of claim 1wherein the historical patient data comprises previously self-reportedpatient well-being status data.
 6. The method of claim 1 wherein theautomatically analyzing comprises applying at least the sensor data to atrained neural network wherein the trained neural network was trainedusing at least the historical patient data.
 7. The method of claim 1wherein the neurological condition of the patient is related to a motordisorder of the patient.
 8. The method of claim 1 wherein theneurological condition of the patient is related to chronic pain of thepatient.
 9. The method of claim 1 wherein the one or more GUI elementsare superimposed over or surrounding bodily regions automaticallyanalyzed for patient movement.
 10. The method of claim 1 wherein the oneor more GUI elements are indicative of tremor of the patient.
 11. Themethod of claim 1 wherein the one or more GUI elements are indicative ofrigidity of the patient.
 12. The method of claim 1 wherein the one ormore GUI elements are modified according to an artificial intelligence(AI) classification of patient movement.
 13. The method of claim 1wherein the one or more GUI elements are modified according to anartificial intelligence (AI) quantification of patient movement.
 14. Themethod of claim 1 wherein the historical patient data comprises patientmedical file data.
 15. The method of claim 14 wherein the historicalpatient data comprises at least one prior medical diagnosis of thepatient.
 16. The method of claim 14 wherein the automatically analyzingcomprises applying at least the sensor data to a trained neural networkwherein the trained neural network was trained for use as apatient-specific model.